Method and apparatus for video coding

ABSTRACT

There are disclosed various methods, apparatuses and computer program products for video encoding and decoding. In some embodiments a method comprises at least one of the following: encoding into a bitstream an indication that motion fields are stored, but only for inter-layer motion prediction; encoding into a bitstream an indication on a limited scope of motion field usage; encoding into a bitstream an indication whether or not to use the motion field for prediction; encoding into a bitstream an indication of storage parameters for storing motion information.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/834,429, filed Mar. 30, 2020, which is a continuation of U.S.application Ser. No. 14/325,615, filed Jul. 8, 2014, which claims thebenefit of U.S. Provisional Application No. 61/843,996, filed Jul. 9,2013, the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present application relates generally to an apparatus, a method anda computer program for video coding and decoding.

BACKGROUND

This section is intended to provide a background or context to theinvention that is recited in the claims. The description herein mayinclude concepts that could be pursued, but are not necessarily onesthat have been previously conceived or pursued. Therefore, unlessotherwise indicated herein, what is described in this section is notprior art to the description and claims in this application and is notadmitted to be prior art by inclusion in this section.

A video coding system may comprise an encoded that transforms an inputvideo into a compressed representation suited for storage/transmissionand a decoder that can uncompress the compressed video representationback into a viewable form. The encoder may discard some information inthe original video sequence in order to represent the video in a morecompact form, for example, to enable the storage/transmission of thevideo information at a lower bitrate than otherwise might be needed.

Various technologies for providing three-dimensional (3D) video contentare currently investigated and developed. Especially, intense studieshave been focused on various multiview applications wherein a viewer isable to see only one pair of stereo video from a specific viewpoint andanother pair of stereo video from a different viewpoint. One of the mostfeasible approaches for such multiview applications has turned out to besuch, wherein only a limited number of input views, e.g. a mono or astereo video plus some supplementary data, is provided to a decoder sideand all required views are then rendered (i.e. synthesized) locally bythe decoder to be displayed on a display.

Some video coding standards introduce headers at slice layer and below,and a concept of a parameter set at layers above the slice layer. Aninstance of a parameter set may include all picture, group of pictures(GOP), and sequence level data such as picture size, display window,optional coding modes employed, macroblock allocation map, and others.Each parameter set instance may include a unique identifier. Each sliceheader may include a reference to a parameter set identifier, and theparameter values of the referred parameter set may be used when decodingthe slice. Parameter sets decouple the transmission and decoding orderof infrequently changing picture, GOP, and sequence level data fromsequence, GOP, and picture boundaries. Parameter sets can be transmittedout-of-band using a reliable transmission protocol as long as they aredecoded before they are referred. If parameter sets are transmittedin-band, they can be repeated multiple times to improve error resiliencecompared to conventional video coding schemes. The parameter sets may betransmitted at a session set-up time. However, in some systems, mainlybroadcast ones, reliable out-of-band transmission of parameter sets maynot be feasible, but rather parameter sets are conveyed in-band inParameter Set NAL units.

SUMMARY

Some embodiments provide a method for encoding and decoding videoinformation. In some embodiments of the present invention there isprovided a method, apparatus and computer program product for videocoding.

Various aspects of examples of the invention are provided in thedetailed description.

According to a first aspect, there is provided a method comprising atleast one of the following:

-   -   a) encoding into a bitstream an indication that motion fields        are stored, but only for inter-layer motion prediction;    -   b) encoding into a bitstream an indication on a limited scope of        motion field usage;    -   c) encoding into a bitstream an indication whether or not to use        the motion field for prediction;    -   d) encoding into a bitstream an indication of storage parameters        for storing motion information.

According to a second aspect, there is provided an apparatus comprisingat least one processor and at least one memory including computerprogram code, the at least one memory and the computer program codeconfigured to, with the at least one processor, cause the apparatus toperform at least one of the following:

-   -   a) encode into a bitstream an indication that motion fields are        stored, but only for inter-layer motion prediction;    -   b) encode into a bitstream an indication on a limited scope of        motion field usage;    -   c) encode into a bitstream an indication whether or not to use        the motion field for prediction;    -   d) encode into a bitstream an indication of storage parameters        for storing motion information.

According to a third aspect there is provided a computer program productincluding one or more sequences of one or more instructions which, whenexecuted by one or more processors, cause an apparatus to at leastperform at least one of the following:

-   -   a) encoding into a bitstream an indication that motion fields        are stored, but only for inter-layer motion prediction;    -   b) encoding into a bitstream an indication on a limited scope of        motion field usage;    -   c) encoding into a bitstream an indication whether or not to use        the motion field for prediction;    -   d) encoding into a bitstream an indication of storage parameters        for storing motion information.

According to a fourth aspect, there is provided a method comprising atleast one of the following:

-   -   a) decoding from a bitstream an indication that motion fields        are stored, but only for inter-layer motion prediction;    -   b) decoding from a bitstream an indication on a limited scope of        motion field usage;    -   c) decoding from a bitstream an indication whether or not to use        the motion field for prediction;    -   d) decoding from a bitstream an indication of storage parameters        for storing motion information.

According to a fifth aspect, there is provided an apparatus comprisingat least one processor and at least one memory including computerprogram code, the at least one memory and the computer program codeconfigured to, with the at least one processor, cause the apparatus toperform at least one of the following:

-   -   a) decode from a bitstream an indication that motion fields are        stored, but only for inter-layer motion prediction;    -   b) decode from a bitstream an indication on a limited scope of        motion field usage;    -   c) decode from a bitstream an indication whether or not to use        the motion field for prediction;    -   d) decode from a bitstream an indication of storage parameters        for storing motion information.

According to a sixth aspect there is provided a computer program productincluding one or more sequences of one or more instructions which, whenexecuted by one or more processors, cause an apparatus to at leastperform at least one of the following:

-   -   a) decoding from a bitstream an indication that motion fields        are stored, but only for inter-layer motion prediction;    -   b) decoding from a bitstream an indication on a limited scope of        motion field usage;    -   c) decoding from a bitstream an indication whether or not to use        the motion field for prediction;    -   d) decoding from a bitstream an indication of storage parameters        for storing motion information.

According to an embodiment, a step a) comprises two or more of thefollowing:

-   -   i. encoding into/decoding from a bitstream an indication whether        or not motion fields are used within a layer for temporal motion        vector prediction;    -   ii. encoding into/decoding from a bitstream an indication        whether or not inter-layer motion prediction is allowed to be        used;    -   iii. encoding into/decoding from a bitsream an indication        whether or not diagonal motion prediction is allowed to be used.

According to an embodiment, in step b) the limited scope defines eithercertain temporal sub-layers or picture types or both.

According to an embodiment, step c) comprises using a specificalgorithms for inferring motion fields to be used for prediction.

According to an embodiment, step c) comprises encoding into/decodingfrom the bitstream a command or a syntax element for controlling motionfield marking.

According to an embodiment, step d) comprises indicating either spatialresolution or accuracy of storing motion information.

According to an embodiment, step d) comprises indicating whichparameters of the motion information are needed in the motionprediction.

According to an embodiments, step d) comprises indicating constraints onparameter of the motion information which reduces the storage space formotion fields.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of example embodiments of the presentinvention, reference is now made to the following descriptions taken inconnection with the accompanying drawings in which:

FIG. 1 shows a block diagram of a video coding system according to anembodiment;

FIG. 2 shows an apparatus for video coding according to an embodiment;

FIG. 3 shows an arrangement for video coding comprising a plurality ofapparatuses;

FIG. 4 shows a high level flow chart of a method according to anembodiment, and

FIG. 5 shows sources of the candidate motion vector predictors.

DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS

In the following, several embodiments of the invention will be describedin the context of one video coding arrangement. It is to be noted,however, that the invention is not limited to this particulararrangement. In fact, the different embodiments have applications widelyin any environment where improvement of non-scalable, scalable and/ormultiview video coding is required. For example, the invention may beapplicable to video coding systems like streaming systems, DVD players,digital television receivers, personal video recorders, systems andcomputer programs on personal computers, handheld computers andcommunication devices, as well as network elements such as transcodersand cloud computing arrangements where video data is handled.

The H.264/AVC standard was developed by the Joint Video Team (JVT) ofthe Video Coding Experts Group (VCEG) of the TelecommunicationsStandardization Sector of International Telecommunication Union (ITU-T)and the Moving Picture Experts Group (MPEG) of InternationalOrganisation for Standardization (ISO)/International ElectrotechnicalCommission (IEC). The H.264/AVC standard is published by both parentstandardization organizations, and it is referred to as ITU-TRecommendation H.264 and ISO/IEC International Standard 14496-10, alsoknown as MPEG-4 Part 10 Advanced Video Coding (AVC). There have beenmultiple versions of the H.264/AVC standard, integrating new extensionsor features to the specification. These extensions include ScalableVideo Coding (SVC) and Multiview Video Coding (MVC).

The H.265/HEVC standard was developed by the Joint Collaborative Team onVideo Coding (JCT-VC) of VCEG and MPEG. The H.265/HEVC standard will bepublished by both parent standardization organizations, and is referredto as ITU-T Recommendation H.265 and ISO/IEC International Standard23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding(HEVC). There are currently ongoing standardization projects to developextensions to H.265/HEVC, including scalable, multiview,three-dimensional, and fidelity range extensions.

A scalable video codec for quality scalability (also known asSignal-to-Noise or SNR) and/or spatial scalability may be implemented asfollows. For a base layer, a conventional non-scalable video encoder anddecoder is used. The reconstructed/decoded pictures of the base layerare included in the reference picture buffer for an enhancement layer.In H.264/AVC, HEVC, and similar codecs using reference picture list(s)for inter prediction, the base layer decoded pictures may be insertedinto a reference picture list(s) for coding/decoding of an enhancementlayer pictures similarly to the decoded reference pictures of theenhancement layer. Consequently, the encoder may choose a base-layerreference picture as inter prediction reference and may indicate its usee.g. with a reference picture index in the coded bitstream. The decoderdecodes from the bitstream, for example from a reference picture index,that a base-layer picture is used as an inter prediction reference forthe enhancement layer. When a decoded base-layer picture is used as aprediction reference for an enhancement layer, it is referred to as aninter-layer reference picture.

Various technologies for providing three-dimensional (3D) video contentare currently investigated and developed. Especially, intense studieshave been focused on various multiview applications wherein a viewer isable to see only one pair of stereo video from a specific viewpoint andanother pair of stereo video from a different viewpoint. One of the mostfeasible approaches for such multiview applications has turned out to besuch wherein only a limited number of views, e.g. a mono or a stereovideo plus some supplementary data, is provided to a decoder side andall required views are then rendered (i.e. synthesized) locally be thedecoder to be displayed on a display.

Some key definitions, bitstream and coding structures, and concepts ofH.264/AVC and HEVC are described in this section as an example of avideo encoder, decoder, encoding method, decoding method, and abitstream structure, wherein the embodiments may be implemented. Some ofthe key definitions, bitstream and coding structures, and concepts ofH.264/AVC are the same as in HEVC—hence, they are described belowjointly. The aspects of the invention are not limited to H.264/AVC orHEVC, but rather the description is given for one possible basis on topof which the invention may be partly or fully realized.

When describing H.264/AVC and HEVC as well as in example embodiments,common notation for arithmetic operators, logical operators, relationoperators, bit-wise operators, assignment operators, and range notatione.g. as specified in H.264/AVC or a draft HEVC may be sued. Furthermore,common mathematical functions e.g. as specified in H.264/AVC or a draftHEVC may be used and a common order or precedence and execution order(from left to right or from right to left) of operators e.g. a specifiedin H.264/AVC or a draft HEVC may be used.

When describing H.264/AVC and HEVC as well as in example embodiments,the following description may be used to specify the parsing process ofeach syntax element.

-   -   b(8): byte having any pattern of bit string (8 bits).    -   se(v): signed integer Exp-Golomb-coded syntax element with the        left bit first.    -   u(n): unsigned integer using n bits. When n is “v” in the syntax        table, the number of bits varies in a manner dependent on the        value of other syntax elements. The paring process for this        descriptor is specified by n next bits from the bitstream        interpreted as a binary representation of an unsigned integer        with the most significant bit written first.    -   ue(v): unsigned integer Exp-Golomb-coded syntax element with the        left bit first.

An Exp-Golomb bit string may be converted to a code number (codeNum) forexample using the following table:

Bit string codeNum 1 0 0 1 0 1 0 1 1 2 0 0 1 0 0 3 0 0 1 0 1 4 0 0 1 1 05 0 0 1 1 1 6 0 0 0 1 0 0 0 7 0 0 0 1 0 0 1 8 0 0 0 1 0 1 0 9 . . . . ..

A code number corresponding to an Exp-Golomb bit string may be convertedto se(v) for example using the following table:

codeNum syntax element value 0 0 1 1 2 −1 3 2 4 −2 5 3 6 −3 . . . . . .

When describing H.264/AVC and HEVC as well as in example embodiments,syntax structures, semantics of syntax elements, and decoding processmay be specified as follows. Syntax elements in the bitstream arerepresented in bold type. Each syntax elements is described by its name(all lower case letters with underscore characteristics), optionally itsone or two syntax categories, and one or two descriptors for its methodof coded representation. The decoding process behaves according to thevalue of the syntax element and to the values of previously decodedsyntax elements. When a value of a syntax element is used in the syntaxtables or the text, named by a mixture of lower case and upper caseletter and without any underscore characters. Variables starting with anupper case letter are derived for the decoding of the current syntaxstructure and all depending syntax structures. Variables starting withan upper case letter may be used in the decoding process for latersyntax structures without mentioning the originating syntax structure ofthe variable. Variables starting with a lower case letter are only usedwithin the context in which they are derived. In some cases, “mnemonic”names for syntax element values or variable values are usedinterchangeably with their numerical values. Sometimes “mnemonic” namesare used without any associated numerical values. The association ofvalues and names is specified in the text. The names are constructedfrom one or more groups of letters separated by an underscore character.Each group starts with an upper case letter and may contain more uppercase letters.

When describing H.264/AVC and HEVC as well as in example embodiments, asyntax structure may be specified using the following. A group ofstatements enclosed in curly brackets is a compound statement and istreated functionally as a single statement. A “while” structurespecifies a test of whether a condition is true, and if true, specifiesevaluation of a statement (or compound statement) repeatedly until thecondition is no longer true. A “do . . . while” structure specifiesevaluation of a statement once, followed by a test of whether acondition is true, and if true, specifies repeated evaluation of thestatement until the condition is no longer true. An “if . . . else”structure specifies a test of whether a condition is true, and if thecondition is true, specifies evaluation of a primary statement,otherwise, specifies evaluation of an alternative statement. The “else”part of the structure and the associated alternative statement isomitted if no alternative statement evaluation is needed. A “for”structure specifies evaluation of an initial statement, followed by atest of a condition, and if the condition is true, specifies repeatedevaluation of a primary statement followed by a subsequent statementuntil the condition is no longer true.

Similarly to many earlier video coding standards, the bitstream syntaxand semantics as well as the decoding process for error-free bitstreamsare specified in H.264/AVC and HEVC. The encoding process is notspecified, but encoders must generate conforming bitstreams. Bitstreamand decoded conformance can be verified with the Hypothetical ReferenceDecoder losses, but the use of the tools in encoding is optional and nodecoding process has been specified for erroneous bitstreams.

The elementary unit for the input to an H.264/AVC or HEVC encoder andthe output of an H.264/AVC or HEVC decoder, respectively, is a picture.A picture may either be a frame or a field. A frame comprises a matrixof luma samples and corresponding chroma samples. A field is a set ofalternate sample rows of a frame and may be used as encoder input, whenthe source signal is interlaced. Chroma pictures may be subsampled whencompared to luma pictures. For example, in the 4:2:0 sampling patternthe spatial resolution of chroma pictures is half of that of the lumapicture along both coordinate axes.

In H.264/AVC, a macroblock is a 16×16 block of luma samples and thecorresponding blocks of chroma samples. For example, in the 4:2:0sampling pattern, a macroblock contains one 8×8 block of chroma samplesper each chroma component. In H.264/AVC, a picture is partitioned to oneor more slice groups, and a slice group contains one or more slices. InH.264/AVC, a slice consists of an integer number of macroblocks orderedconsecutively in the raster scan within a particular slice group.

In a draft HEVC standard, video pictures are divided into coding units(CU) covering the area of the picture. A CU consists of one or moreprediction units (PU) defining the prediction process for the sampleswithin the CU and one or more transform units (TU) defining theprediction error coding process for the samples in the CU. Typically, aCU consists of a square block of samples with a size selectable from apredefined set of possible CU sizes. A CU with the maximum allowed sizeis typically named as CTU (coding tree unit) and the video picture isdivided into non-overlapping CTUs. An CTU can be further split into acombination of smaller CUs, e.g. by recursively splitting the CTU andresultant CUs. Each resulting CU typically has at least one PU and atleast on TU associated with it. Each PU and TU can further be split intosmaller PUs and TUs in order to increase granularity of the predictionand prediction error coding processes, respectively. The PU splittingcan be realized by splitting the CU into four equal size square PUs orsplitting the CU into two rectangle PUs vertically or horizontally in asymmetric or asymmetric way. The division of the image into CUs, anddivision of CUs into PUs and TUs is typically signaled in the bitstreamallowing the decoder to reproduce the intended structure of these units

In a draft HEVC standard, a picture can be partitioned in tiles, whichare rectangular and contain an integer number of CTUs. In the currentworking draft of HEVC, the partitioning to tiles forms a regular grid,where heights and widths of tiles differ from each other by one CTU atthe maximum. In a draft HEVC, a slice consists of an integer number ofCUs. The CUs are scanned in the raster scan order of CTUs within tilesor within a picture, if tiles are not in use. Within an CTU, the CUshave a specific scan order.

In a Working Draft (WD) 5 of HEVC, some key definitions and concepts forpicture partitioning are defined as follows. A partitioning is definedas the division of a set into subsets such that each element of the setis in exactly one of the subsets.

A basic coding unit in a HEVC WD5 is a treeblock. A treeblock is an N×Nblock of luma samples and two corresponding blocks of chroma samples ofa picture that has three sample arrays, or an N×N block of samples of amonochrome picture or a picture that is coded using three separatecolour planes. A treeblock may be partitioned for different coding anddecoding processes. A treeblock partition is a block of luma samples andtwo corresponding blocks of chroma samples resulting from a partitioningof a treeblock for a picture that has three sample arrays or a block ofluma samples resulting from a partitioning of a treeblock for amonochrome picture or a picture that is coded using three separatecolour planes. Each treeblock is assigned a partition signalling toidentify the block sizes for intra or inter prediction and for transformcoding. The partitioning is a recursive quadtree partitioning. The rootof the quadtree is associated with the treeblock. The quadtree is splituntil a leaf is reached, which is referred to as the coding node. Thecoding node is the root node of two tress, the prediction tree and thetransform tree. The prediction tree specifies the position and size ofprediction blocks. The prediction tree and associated prediction dataare referred to as a prediction unit. The transform tree specifies theposition and size of transform blocks. The transform tree and associatedtransform data are referred to as a transform unit. The splittinginformation for luma and chroma is identical for the prediction tree andmay or may not be identical for the transform tree. The coding node andthe associated prediction and transform units form together a codingunit.

In a HEVC WD5, pictures are divided into slices and tiles. A slice maybe a sequence of treeblocks but (when referring to a so-called finegranular slice) may also have its boundary within a treeblock at alocation where a transform unit and prediction unit coincide. Treeblockswithin a slice are coded and decoded in a raster scan order. For theprimary coded picture, the division of each picture into slices is apartitioning.

In a HEVC WD5, a tile is defined as an integer number of treeblocksco-occurring in one column and one row, ordered consecutively in theraster scan within the tile. For the primary coded picture, the divisionof each picture into tiles is a partitioning. Tiles are orderedconsecutively in the raster scan within the picture. Although a slicecontains treeblocks that are consecutive in the raster scan within atile, these treeblocks are not necessarily consecutive in the rasterscan within the picture. Slices and tiles need not contain the samesequence of treeblocks. A tile may comprise treeblocks contained in morethan one slice. Similarly, a slice may comprises treeblocks contained inseveral tiles.

In H.264/AVC and HEVC, in-picture prediction may be disabled acrossslice boundaries. Thus, slices can be regarded as a way to split a codedpicture into independently decodable pieces, and slices are therefore ofthe regarded as elementary units for transmission. In many cases,encoders may indicate in the bitstream which types of in-pictureprediction are turned off across slice boundaries, and the decoderoperation takes this information into account for example whenconcluding which prediction sources are available. For example, samplesfrom a neighboring macroblock or CU may be regarded as unavailable forintra prediction, if the neighboring macroblock or CU resides in adifferent slice.

A syntax element may be defined as an element of data represented in thebitstream. A syntax structure may be defined as zero or more syntaxelements present together in the bitstream in a specified order.

The elementary unit for the output of an H.264/AVC or HEVC encoder andthe input of an H.264/AVC or HEVC decoder respectively, is a NetworkAbstraction Layer (NAL) unit. For transport over packet-orientednetworks or storage into structured files, NAL units may be encapsulatedinto packets or similar structures. A bytestream format has beenspecified in H.264/AVC and HEVC for transmission or storage environmentsthat do not provide framing structures. The bytestream format separatesNAL units from each other by attaching a start code in front of each NALunit. To avoid false dection of NAL unit boundaries, encoders may run abyte-oriented start code emulation prevention algorithm, which adds anemulation prevention byte to the NAL unit payload if a start code wouldhave occurred otherwise. In order to enable straightforward gatewayoperation between packet- and stream-oriented systems, start codeemulation prevention may always be performed regardless of whether thebytestream format is in use or not.

NAL units consist of a header and payload. In H.264/AVC, the NAL unitheader indicates the type of the NAL unit and whether a coded slicecontained in the NAL unit is a part of a reference picture or anon-reference picture. H.264/AVC includes a 2-bit nal_ref_idc syntaxelement, which when equal to 0 indicates that a coded slice contained inthe NAL unit is a part of a non-reference picture and when greater than0 indicates that a coded slice contained in the NAL unit is a part of areference picture. The header for SVC and MVC NAL units may beadditionally contain various indications related to scalability andmultiview hierarchy.

In HEVC, a two-byte NAL unit header is used for all specified NAL unittypes. The NAL unit header contains one reserved bit, a six-bit NAL unittype indication, a six-bit reserved field (called nuh_layer_id) and athree-bit temporal_id_plus1 indication for temporal level. Thetemporal_id_plus1 syntax element may be regarded as a temporalidentifier for the NAL unit, and a zero-based TemporalId variable may bederived as follows: TemporalId=temporal_id_plus1−1. TemporalId equal to0 corresponds to the lowest temporal level. The value oftemporal_id_plus1 is required to be non-zero in order to avoid startcode emulation involving the two NAL unit header bytes. The bitstreamcreated by excluding all VCL NAL units having a TemporalId greater thanor equal to a selected value and including all other VCL NAL unitsremains conforming. Consequently, a picture having TemporalId equal toTID does not use any picture having a TemporalId greater than TID asinter prediction reference. A sub-layer or a temporal sub-layer may bedefined to be a temporal scalable layer of a temporal scalablebitstream, consisting of VCL NAL units with a particular value of theTemporalId variable and the associated non-VCL NAL units. Without lossof generality, in some example embodiments a variable LayerId is derivedfrom the value of nuh_layer_id for example as follows:LayerId=nuh_layer_id. In the following, LayerId, nuh_layer_id andlayer_id are used interchangeably unless otherwise indicated.

It is expected that nuh_layer_id and/or similar syntax elements in NALunit header would carry information on the scalability hierarchy. Forexample, the LayerId value may be mapped to values of variables orsyntax elements describing different scalability dimensions, such asquality_id or similar, dependency_id or similar, any other type of layeridentifier, view order index or similar, view identifier, an indicationwhether the NAL unit concerns depth or texture i.e. depth_flag orsimilar, or an identifier similar to priority_id of SVC indicating avalid sub-bitstream extraction if all NAL units greater than a specificidentifier value are removed from the bitstream. nuh_layer_id and/orsimilar syntax elements may be partitioned into one or more syntaxelements indicating scalability properties. For example, a certainnumber of bits among nuh_layer_id and/or similar syntax elements may beused for dependency_id or similar, while another certain number of bitsamong nuh_layer_id and/or similar syntax elements may be used forquality_id or similar. Alternatively, a mapping of LayerId values orsimilar to values of variables or syntax elements describing differentscalability dimensions may be provided for example in a Video ParameterSet, a Sequence Parameter Set or another syntax structure.

NAL units can be categorized into Video Coding Layer (VCL) NAL units andnon-VCL NAL units. VCL NAL units are typically coded slice NAL units. InH.264/AVC, coded slice NAL units contain syntax elements representingone or more coded macroblocks, each of which corresponds to a block ofsamples in the uncompressed picture. In HEVC, coded slice NAL unitscontain syntax elements representing one or more CU.

In H.264/AVC, a coded slice NAL unit can be indicated to be a codedslice in an Instantaneous Decoding Refresh (IDR) picture or coded slicein a non-IDR picture.

In a draft HEVC standard, a coded slice NAL unit can be indicated to beone of the following types:

Name of Content of NAL unit and RBSP nal_unit_type nal_unit_type syntaxstructure 0, TRAIL_N, Coded slice segment of a non-TSA, 1 TRAIL_Rnon-STSA trailing picture slice_segment_layer_rbsp( ) 2, TSA_N, Codedslice segment of a TSA picture 3 TSA_R slice_segment_layer_rbsp( ) 4,STSA_N, Coded slice segment of an STSA 5 STSA_R pictureslice_layer_rbsp( ) 6, RADL_N, Coded slice segment of a RADL 7 RADL_Rpicture slice_layer_rbsp( ) 8, RASL_N, Coded slice segment of a RASL 9RASL_R, picture slice_layer_rbsp( ) 10, RSV_VCL_N10 Reserved // reservednon-RAP non- 12, RSV_VCL_N12 reference VCL NAL unit types 14 RSV_VCL_N1411, RSV_VCL_R11 Reserved // reserved non-RAP 13, RSV_VCL_R13 referenceVCL NAL unit types 15 RSV_VCL_R15 16, BLA_W_LP Coded slice segment of aBLA picture 17, BLA_W_DLP slice_segment_layer_rbsp( ) 18 BLA_N_LP 19,IDR_W_DLP Coded slice segment of an IDR 20 IDR_N_LP pictureslice_segment_layer_rbsp( ) 21 CRA_NUT Coded slice segment of a CRApicture slice_segment_layer_rbsp( ) 22, RSV_RAP_VCL22.. Reserved //reserved RAP VCL NAL 23 RSV_RAP_VCL23 unit types 24..31 RSV_VCL24..Reserved // reserved non-RAP VCL RSV_VCL31 NAL unit types

In a draft HEVC standard, abbreviations for picture types may be definedas follows: trailing (TRAIL) picture, Temporal Sub-layer Access (TSA),Step-wise Temporal Sub-layer Access (STSA), Random Access DecodableLeading (RADL) picture, Random Access Skipped Leading (RASL) picture,Broken Link Access (BLA) picture, Instantaneous Decoding Refresh (IDR)picture, Clean Random Access (CRA) picture.

A Random Access Point (RAP) picture, which may also or alternatively bereferred to as intra random access point (TRAP) picture, is a picturewhere each slice or slice segment has nal_unit_type in the range of 16to 23, inclusive. A RAP picture contains only intra-coded slices, andmay be a BLA picture, a CRA picture or an IDR picture. The first picturein the bitstream is a RAP picture. Provided the necessary parameter setsare available when they need to be activated, the RAP picture and allsubsequent non-RASL pictures in decoding order can be correctly decodedwithout performing the decoding process of any pictures that precede theRAP picture in decoding order. There may be pictures in a bitstream thatcontain only intra-coded slices that are not RAP pictures.

In HEVC a CRA picture may be the first picture in the bitstream indecoding order, or may appear later in the bitstream. CRA pictures inHEVC allow so-called leading pictures that follow the CRA picture indecoding order but precede it in output order. Some of the leadingpictures, so-called RASL pictures, may use pictures decoded before theCRA picture as a reference. Pictures that follow a CRA picture in bothdecoding and output order are decodable if random access is performed atthe CRA picture, and hence clean random access is achieved similarly tothe clean random access functionality of an IDR picture.

A CRA picture may have associated RADL or RASL pictures. When a CRApicture is the first picture in the bitstream in decoding order, the CRApicture is the first picture of a coded video sequence in decodingorder, and any associated RASL pictures are not output by the decoderand may not be decodable, as they may contain references to picturesthat are not present in the bitstream.

A leading picture is a picture that precedes the associated RAP picturein output order. The associated RAP picture is the previous RAP picturein decoding order (if present). A leading picture may either be a RADLpicture or a RASL picture.

All RASL pictures are leading pictures of an associated BLA or CRApicture. When the associated RAP picture is a BLA picture or is thefirst coded picture in the bitstream, the RASL picture is not output andmay not be correctly decodable, as the RASL picture may containreferences to pictures that are not present in the bitstream. However, aRASL picture can be correctly decoded if the decoding had started from aRAP picture before the associated RAP picture of the RASL picture. RASLpictures are not used as reference pictures for the decoding process ofnon-RASL pictures. When present, all RASL pictures precede, in decodingorder, all trailing pictures of the same associated RAP picture. In someearlier drafts of the HEVC standard, a RASL picture was referred to aTagged for Discard (TFD) picture.

All RADL pictures are leading pictures. RADL pictures are not used asreference pictures for the decoding process of trailing pictures of thesame associated RAP picture. When present, all RADL pictures precede, indecoding order, all trailing pictures of the same associated RAPpicture. RADL pictures do not refer to any picture preceding theassociated RAP picture in decoding order and can therefore be correctlydecoded when the decoding starts from the associated RAP picture. Insome earlier drafts of the HEVC standard, a RADL picture was referred toa Decodable Leading Picture (DLP).

Decodable leading pictures may be such that can be correctly decodedwhen the decoding is started from the CRA picture. In other words,decodable leading pictures use only the initial CRA picture orsubsequent pictures in decoding order as reference in inter prediction.Non-decodable leading pictures are such that cannot be correctly decodedwhen the decoding is started from the initial CRA picture. In otherwords, non-decodable leading pictures use pictures prior, in decodingorder, to the initial CRA picture as references in inter prediction.

When a part of a bitstream starting from a CRA picture is included inanother bitstream, the RASL pictures associated with the CRA picturemight not be correctly decodable, because some of their referencepictures might not be present in the combined bitstream. To make such asplicing operation straightforward, the NAL unit type of the CRA picturecan be changed to indicate that it is a BLA picture. The RASL picturesassociated with a BLA picture may not be correctly decodable hence arenot be output/displayed. Furthermore, the RASL pictures associated witha BLA picture may be omitted from decoding.

A BLA picture may be the first picture in the bitstream in decodingorder, or may appear later in the bitstream. Each BLA picture begins anew coded video sequence, and has similar effect on the decoding processas an IDR picture. However, a BLA picture contains syntax elements thatspecify a non-empty reference picture set. When a BLA picture hasnal_unit_type equal to BLA_W_LP, it may have associated RASL pictures,which are not output by the decoder and may not be decodable, as theymay contain references to pictures that are not present in thebitstream. When a BLA picture has nal_unit_type equal to BLA_W_LP, itmay also have associated RADL pictures, which are specified to bedecoded. When a BLA picture has nal_unit_type equal to BLA_W_DLP, itdoes not have associated RASL pictures but may have associated RADLpictures, which are specified to be decoded. BLA_W_DLP may also bereferred to as BLA_W_RADL. When a BLA picture has nal_unit_type equal toBLA_N_LP, it does not have any associated leading pictures.

An IDR picture having nal_unit_type equal to IDR_N_LP does not haveassociated leading pictures present in the bitstream. An IDR picturehaving nal_unit_type equal to IDR_W_DLP does not have associated RASLpictures present in the bitstream, but may have associated RADL picturesin the bitstream. IDR_W_DLP may also be referred to as IDR_W_RADL.

When the value of nal_unit_type is equal to TRAIL_N, TSA_N, STSA_N,RADL_N, RASL_N, RSV_VCL_N10, RSV_VCL_N12, or RSV_VCL_N14, the decodedpicture is not used as a reference for any other picture of the sametemporal sub-layer. That is, in a draft HEVC standard, when the value ofnal_unit_type is equal to TRAIL_N, TSA_N, STSA_N, RADL_N, RASL_N,RSV_VCL_N10, RSV_VCL_N12, or RSV_VCL_N14, the decoded picture is notincluded in any of RefPicSetStCurrBefore, RefPicSetStCurrAfter andRefPicSetLtCurr of any picture with the same value of TemporalId. Acoded picture with nal_unit_type equal to TRAIL_N, TSA_N, STSA_N,RADL_N, RASL_N, RSV_VCL_N10, RSV_VCL_N12, or RSV_VCL_N14 may bediscarded without affecting the decodability of other pictures with thesame value of TemporalId.

A trailing picture may be defined as a picture that follows theassociated RAP picture in output order. Any picture that is a trailingpicture does not have nal_unit_type equal to RADL_N, RADL_R, RASL_N orRASL_R. Any picture that is a leading picture may be constrained toprecede, in decoding order, all trailing pictures that are associatedwith the same RAP picture. No RASL pictures are present in the bitstreamthat are associated with a BLA picture having nal_unit_type equal toBLA_W_DLP or BLA_N_LP. No RADL pictures are present in the bitstreamthat are associated with a BLA picture having nal_unit_type equal toBLA_N_LP or that are associated with an IDR picture having nal_unit_typeequal to IDR_N_LP. Any RASL picture associated with a CRA or BLA picturemay be constrained to precede any RADL picture associated with the CRAor BLA picture in output order. Any RASL picture associated with a CRApicture may be constrained to follow, in output order, any other RAPpicture that precedes the CRA picture in decoding order.

In HEVC, there are two picture types, the TSA and STSA picture types,that can be used to indicate temporal sub-layer switching points. Iftemporal sub-layers with TemporalId up to N had been decoded until theTSA or STSA picture (exclusive) and the TSA or STSA picture hasTemporalId equal to N+1, the TSA or STSA picture enables decoding of allsubsequent pictures (in decoding order) having TemporalId equal to N+1.The TSA picture type may impose restrictions on the TSA picture itselfand all pictures in the same sub-layer that follow the TSA picture indecoding order. None of these pictures is allowed to use interprediction from any picture in the same sub-layer that precedes the TSApicture in decoding order. The TSA definition may further imposerestrictions on the pictures in higher sub-layers that follow the TSApicture in decoding order. None of these pictures is allowed to refer apicture that precedes the TSA picture in decoding order if that picturebelongs to the same or higher sub-layer as the TSA picture. TSA pictureshave TemporalId greater than 0. The STSA is similar to the TSA picturebut does not impose restrictions on the pictures in higher sub-layersthat follow the STSA picture in decoding order and hence enableup-switching only onto the sub-layer where the STSA picture resides.

A non-VCL NAL unit may be for example one of the following types: asequence parameter set, a picture parameter set, a supplementalenhancement information (SEI) NAL unit, an access unit delimiter, an endof sequence NAL unit, an end of stream NAL unit, or a filler data NALunit. Parameter sets may be needed for the reconstruction of decodedpictures, whereas many of the other non-VCL NAL units are not necessaryfor the reconstruction of decoded sample values.

Parameters that remain unchanged through a coded video sequence may beincluded in a sequence parameter set (SPS). In addition to theparameters that may be essential to the decoding process, the sequenceparameter set may optionally contain video usability information (VUI),which includes parameters that may be important for buffering, pictureoutput timing, rendering and resource reservation. There are three NALunits specified in H.264/AVC to carry sequence parameter sets: thesequence parameter set NAL unit containing all the data for H.264/AVCVCL NAL units in the sequence, the sequence parameter set extension NALunit containing the data for auxiliary coded pictures, and the subsetsequence parameter set for MVC and SVC VCL NAL units. A pictureparameter set (PPS) contains such parameters that are likely to beunchanged in several coded pictures.

Parameter set syntax structures may have extensions mechanisms, whichmay for example be used to include parameters that are specific toextensions of a coding standard. An example syntax of an extensionmechanism is provided in the following for SPS:

Descriptor seq_parameter_set_rbsp( ) {  ...  sps_extension_flag u(1) if( sps_extension_flag )   while( more_rbsp_data( ) )   sps_extension_data_flag u(1)  rbsp_trailing_bits( ) }

Decoders of particular version(s) of a coding standard or a codingscheme may ignore sps_extension_dataflag, while in another version ofthe coding standard or the coding scheme, an extension syntax structuremay be specified and may appear within the sps_extension_data_flag bits.Similar extensions mechanisms may be specifies also for other types ofparameter sets.

In a draft HEVC, there is also a third type of parameter sets, herereferred to as Adaptation Parameter Set (APS), which includes parametersthat are likely to be unchanged in several coded slices. In a draftHEVC, the APS syntax structure includes parameters or syntax elementsrelated to context-based adaptive binary arithmetic coding (CABAC),adaptive sample offset, adaptive loop filtering, and deblockingfiltering. In a draft HEVC, an APS is a NAL unit and coded withoutreference or prediction from any other NAL unit. An identifier, referredto as aps_id syntax element, is included in APS NAL unit, and includedand used in the slice header to refer to a particular APS. However, APSwas not included in the final H.265/HEVC standard.

H.265/HEVC also includes another type of a parameter set, called a videoparameter set (VPS). A video parameter set RBSP may include parametersthat can be referred to by one or more sequence parameter set RBSPs.

The relationship and hierarchy between VPS, SPS, and PPS may bedescribed as follows. VPS resides one level above SPS in the parameterset hierarchy and in the context of scalability and/or 3DV. VPS mayinclude parameters that are common for all slices across all(scalability or view) layers in the entire coded video sequence. SPSincludes the parameters that are common for all slices in a particular(scalability or view) layer in the entire coded video sequence, and maybe shared by multiple (scalability or view) layers. PPS includes theparameters that are common for all slices in a particular layerrepresentation (the representation of one scalability or view layer inone access unit) and are likely to be shared by all slices in multiplelayer representations.

VPS may provide information about the dependency relationships of thelayers in a bitstream, as well as many other information that areapplicable to all slices across all (scalability or view) layers in theentire coded video sequence. In a scalable extension of HEVC, VPS mayfor example include a mapping of the LayerId value derived from the NALunit header to one or more scalability dimension values, for examplecorrespond to dependency_id, quality_id, view_id, and depth_flag for thelayer defined similarly to SVC and MVC. VPS may include profile andlevel information for one or more layers as well as the profile and/orlevel for one or more temporal sub-layers (consisting of VCL NAL unitsat and below certain TemporalId values) of a layer representation. VPSmay also provide the maximum number of layers present in the bitstream.For example, the syntax element vps_max_layers_minus1 may be included inthe syntax and vps_max_layer_minus1+1 may indicate the maximum number oflayers present in the bitstream. The actual number of layers in thebitstream may be smaller than or equal to vps_max_layer_minus1+1.

An example syntax of a VPS extension intended to be a part of the VPS isprovided in the following. The presented VPS extension provides thedependency relationships among other things. It should be understoodthat the VPS extension syntax is provided as an example and othersimilar and/or extended syntax structures may be equivalently appliedwith different embodiments.

Descriptor vps_extension( ) {  while( !byte_aligned( ) )  vps_extension_byte_alignment_reserved_one_bit u(1)  for( i = 0,NumScalabilityTypes = 0; i < 16; i++) {   scalability_mask[ i ] u(1)  NumScalabilityTypes += scalability_mask[ i ]  }  for( j = 0; j <NumScalabilityTypes; j++)   dimension_id_len_minus1[ j ] u(3) vps_nuh_layer_id_present_flag u(1)  for( i = 0; i +=vps_max_layers_minus1; i++) {   if( vps_nuh_layer_id_present_flag &&i >0)    layer_id_in_nuh[ i ] u(6)   for( j = 0; j <NumScalabilityTypes; j++)    dimension_id[ i ][ j ] u(v)  }  for( i = 1;i <= vps_max_layers_minus1; i++)   for( j = 0; j < i; j++)   direct_dependency_flag[ i ][ j ] u(1)  direct_dep_type_len_minus2ue(v)  for( i = 1; i <= vps_max_layers_minus1; i++)   for( j = 0; j < i;j++)    if( direct_dependency_flag[ i ][ j ] )    direct_dependency_type[ i ][ j ] u(v) }

The semantics of the presented VPS extension may be specified asdescribed in the following paragraphs.

vps_extension_byte_alignment_reserved_one_bit is equal to 1 and is usedto achieve alignment of the next syntax element to a byte boundary.

scalability_mask[i] equal to 1 indicates that dimension_id syntaxelements corresponding to the i-th scalability dimension are present.scalability_mask[i] equal to 0 indicates that dimension_id syntaxelements corresponding to the i-th scalability dimension are notpresent. The scalability dimensions corresponding to each value of i inscalability_mask[i] may be specified for example to include thefollowing or any subset thereof along with other scalability dimensions.

scalability_mask Scalability ScalabilityId index dimension mapping 0multiview ViewId 1 spatial or quality DependencyId scalability

dimension_id_len_minus1[j] plus 1 specifies the length, in bits, of thedimension_id[i][j] syntax element. vps_nuh_layer_id_present_flagspecifies whether the layer_id_in_nuh[i] syntax is present.layer_id_in_nuh[i] specifies the value of the nuh_layer_id syntaxelement in VCL NAL units of the i-th layer. When not present, the valueof layer_id_in_nuh[i] is inferred to be equal to i. layer_id_in_nuh[i]is greater than layer_id_in_nuh[i−1]. The variableLayerIdxInVps[layer_id_in_nuh[i]] is set equal to i. dimension_id[i][j]specifies the identifier of the j-th scalability dimension type of thei-th layer. When not present, the value of dimension_id[i][j] isinferred to be equal to 0. The number of bits used for therepresentation of dimension_id[i][j] is dimension_id_len_minus1 [j]+1bits.

direct_dependency_flag[i][j] equal to 0 specifies that the layer withindex j is not a direct reference layer for the layer with index i.direct_dependency_flag[i][j] equal to 1 specifies that the layer withindex j may be a direct reference layer for the layer with index i. Whendirect_dependency_flag[i][j] is not present for i and j in the range of0 to vpsmax_num_layers_minus1, it is inferred to be equal to 0.

The variables NumDirectRefLayers[i] and RefLayerId[i][j] may be derivedas follows:

for( i = 1; i <= vps_max_layers_minus1; i++ )  for( j = 0,NumDirectRefLayers[ i ] = 0; j < i; j++ )   if( direct_dependency_flag[i ][ j ] = = 1 )    RefLayerId[ i ][ NumDirectRefLayers[ i ]++ ] =layer_id_in_    nuh[ j ]

direct_dep_type_len_minus2 plus 2 specifies the number of bits of thedirect_dependency_type[i][j] syntax element.direct_dependency_type[i][j] equal to 0 indicates that sample predictionmay be used and motion prediction is not used for layer identified by ifrom layer identified by j. direct_dependency_type[i][j] equal to 1indicates that motion prediction may be used and sample prediction isnot used for layer identified by i from layer identified by j.direct_dependency_type[i][j] equal to 2 indicates that both sample andmotion prediction may be used for layer identified by i from layeridentified by j.

The variables NumSamplePredRefLayers[i], NumMotionPredRefLayers[i],SamplePredEnabledFlag[i][j], MotionPredEnabledFlag[i][j],NumDirectRefLayers[i], RefLayerId[i][j], MotionPredRefLayerId[i][j], andSamplePredRefLayerId[i][j] may be derived as follows:

for( i = 0; i < 64; i++ ) {  NumSamplePredRefLayers[ i ] = 0 NumMotionPredRefLayers[ i ] = 0  NumDirectRefLayers[ i ] = 0  for( j =0; j < 64; j++ ) {   SamplePredEnabledFlag[ i ][ j ] = 0  MotionPredEnabledFlag[ i ][ j ] = 0   RefLayerId[ i ][ j ] = 0  SamplePredRefLayerId[ i ][ j ] = 0   MotionPredRefLayerId[ i ][ j ] =0  } } for( i = 1; i <= vps_max_layers_minus1; i++ ) {  iNuhLId =layer_id_in_nuh[ i ]  for( j = 0; j < i; j++ )   if(direct_dependency_flag[ i ][ j ] ) {     RefLayerId[ iNuhLId ][NumDirectRefLayers[ iNuhLId ]++ ] = layer_id_in_nuh[ j ]   SamplePredEnabledFlag[ iNuhLId ][ j ] = ( ( direct_dependency_type[ i][ j ] + 1 ) & 1 )    NumSamplePredRefLayers[ iNuhLId ] +=SamplePredEnabledFlag[ iNuhLId ][ j ]    MotionPredEnabledFlag[ iNuhLId][ j ] = ( ( ( direct_dependency_type[ i ][ j ] + 1 ) & 2 ) >> 1 )   NumMotionPredRefLayers[ iNuhLId ] += MotionPredEnabledFlag[ iNuhLId][ j ]   } } for( i = 1, mIdx = 0, sIdx = 0; i <= vps_max_layers_minus1;i++ ) {  iNuhLId = layer_id_in_nuh[ i ]  for( j = 0, j < i; j++ ) {  if( MotionPredEnabledFlag[ iNuhLId ][ j ] )    MotionPredRefLayerId[iNuhLId ][ mIdx++ ] = layer_id_in_nuh[ j ]   if( SamplePredEnabledFlag[INuhLid ][ j ] )    SamplePredRefLayerId[ iNuhLid ][ sIdx++ ] =layer_id_in_nuh[ j ]  } }

H.264/AVC and HEVC syntax allows many instances of parameter sets, andeach instance is identified with a unique identifier. In H.264/AVC, eachslice header includes the identifier of the picture parameter set thatis active for the decoding of the picture that contains the slice, andeach picture parameter set contains the identifier of the activesequence parameter set. Consequently, the transmission of picture andsequence parameter set does not have to be accurately synchronized withthe transmission of slices. Instead, it is sufficient that the activesequence and picture parameter sets are received at any moment beforethey are referenced, which allows transmission of parameter sets“out-of-band” using a more reliable transmission mechanism compared tothe protocols used for the slice data. For example, parameter sets canbe included as a parameter in the session description for Real-timeTransport Protocol (RTP) sessions. If parameter sets are transmittedin-band, they can be repeated to improve error robustness.

A SEI NAL unit may contain one or more SEI message, which are notrequired for the decoding of output pictures but assist in relatedprocesses, such as picture output timing, rendering, error detection,error concealment, and resource reservation. Several SEI messages arespecified in H.264/AVC and HEVC, and the user data SEI messages enableorganizations and companies to specify SEI messages for their own use.H.264/AVC and HEVC contain the syntax and semantics for the specifiedSEI messages but no process for handling the messages in the recipientis defined. Consequently, encoders are required to follow the H.264/AVCstandard or the HEVC standard when they create SEI messages, anddecoders conforming to the H.264/AVC standard or the HEVC standard,respectively, are not required to process SEI messages for output orderconformance. One of the reasons to include the syntax and semantics ofSEI messages in H.264/AVC and HEVC is to allow different systemspecifications to interpret the supplemental information identically andhence interoperate. It is intended that system specifications canrequire the use of particular SEI messages both in the encoding end andin the decoding end, and additionally the process for handlingparticular SEI messages in the recipient can be specified.

A coded picture is a coded representation of a picture. A coded picturein H.264/AVC comprises the VCL NAL units that are required for thedecoding of the picture. In H.264/AVC, a coded picture can be a primarycoded picture or a redundant coded picture. A primary coded picture isused in the decoding process of valid bitstreams, whereas a redundantcoded picture is a redundant representation that should only be decodedwhen the primary coded picture cannot be successfully decoded. In adraft HEVC, no redundant coded picture has been specified.

In H.264/AVC and HEVC, an access unit comprises a primary coded pictureand those NAL units that are associated with it. In HEVC, an access unitis defined as a set of NAL units that are associated with each otheraccording to a specified classification rule, are consecutive indecoding order, and contain exactly one coded picture. In H.264/AVC, theappearance order of NAL units within an access unit is constrained asfollows. An optional access unit delimiter NAL unit may indicate thestart of an access unit. It is followed by zero or more SEI NAL units.The coded slices of the primary coded picture appear next. In H.264/AVC,the coded slice of the primary coded picture may be followed by codedslices for zero or more redundant coded pictures. A redundant codedpicture is a coded representation of a picture or a part of a picture. Aredundant coded picture may be decoded if the primary coded picture isnot received by the decoder for example due to the loss in transmissionor a corruption in physical storage medium.

In H.264/AVC, an access unit may also include an auxiliary codedpicture, which is a picture that supplements the primary coded pictureand may be used for example in the display process. An auxiliary codedpicture may for example be used as an alpha channel or alpha planespecifying the transparency level of the samples in the decodedpictures. An alpha channel or plane may be used in a layered compositionor rendering system, where the output picture is formed by overlayingpictures being at least partly transparent on top of each other. Anauxiliary coded picture has the same syntactic and semantic restrictionsas a monochrome redundant coded picture. In H.264/AVC, an auxiliarycoded picture contains the same number of macroblocks as the primarycoded picture.

In HEVC, an access unit may be defined as a set of NAL units that areassociated with each other according to a specified classification rule,are consecutive in decoding order, and contain exactly one codedpicture. In addition to containing the VCL NAL units of the codedpicture, an access unit may also contain non-VCL-NAL units. In HEVC, thedecoding of an access unit results in a decoded picture.

In H.264/AVC, a coded video sequence is defined to be a sequence ofconsecutive access units in decoding order from an IDR access unit,inclusive, to the next IDR access unit, exclusive, or to the end of thebitstream, whichever appears earlier. In HEVC, a coded video sequence isdefined to be a sequence of access units that consists, in decodingorder, of a CRA (Clean Random Access) access unit that is the firstaccess unit in the bitstream, and IDR access unit or a BLA (Broken LinkAccess) access unit, followed by zero or more non-IDR and non-BLA accessunits including all subsequent access units up to but no including anysubsequent IDR or BLA access unit.

A group of pictures (GOP) and its characteristics may be defined asfollows. A GOP can be decoded regardless of whether any previouspictures were decoded. An open GOP is such a group of pictures in whichpictures preceding the initial intra picture in output order might notbe correctly decodable when the decoding starts from the initial intrapicture of the open GOP. In other words, pictures of an open GOP mayrefer (in inter prediction) to pictures belonging to a previous GOP. AnH.264/AVC decoder can recognize an intra picture starting an open GOPfrom the recovery point SEI message in an H.264/AVC bitstream. An HEVCdecoder can recognize an intra picture starting an open GOP, because aspecific NAL unit type, CDR NAL unit type, is used for its coded slices.A closed GOP is such a group of pictures in which all pictures can becorrectly decoded when the decoding starts from the initial intrapicture of the closed GOP. In other words, no picture in a closed GOPrefers to any pictures in previous GOPs. In H.264/AVC and HEVC, a closedGOP starts from an IDR access unit. In HEVC a closed GOP may also startfrom BLA W DLP or a BLA N LP picture. As a result, closed GOP structurehas more error resilience potential in comparison to the open GOPstructure, however at the cost of possible reduction in the compressionefficiency. Open GOP coding structure is potentially more efficient inthe compression, due to a larger flexibility in selection of referencepictures.

The bitstream syntax of H.264/AVC and HEVC indicates whether aparticular picture is a reference picture for inter prediction of anyother picture. Pictures of any coding type (I, P, B) can be referencepictures or non-reference pictures in H.264/AVC and HEVC. The NAL unitheader indicates the type of the NAL unit and whether a coded slicecontained in the NAL unit is a part of a reference picture or anon-reference picture.

Many hybrid video codecs, including H.264/AVC and HEVC, encode videoinformation in two phases. In the first phase, predictive coding isapplied for example as so-called sample prediction and/or so-calledsyntax prediction.

In the sample prediction, pixel or sample values in a certain picturearea or “block” are predicted. These pixel or sample values can bepredicted, for example, using one or more of the following ways:

-   -   Motion compensation mechanisms (which may also be referred to as        temporal prediction or motion-compensated temporal prediction or        motion-compensated prediction or MCP), which involve finding and        indicating an area in one of the previously encoded video frames        that corresponds closely to the block being coded.    -   Inter-view prediction, which involves finding and indicating an        area in one of the previously encoded view components that        corresponds closely to the block being coded.    -   View synthesis prediction, which involves synthesizing a        prediction block or image area where a prediction block is        derived on the basis of reconstructed/decoded ranging        information.    -   Inter-layer prediction using reconstructed/decoded samples, such        as the so-called IntraBL (base layer) mode of SVC.    -   Inter-layer residual prediction, in which for example the coded        residual of a reference layer or a derived residual from a        difference of a reconstructed/decoded reference layer picture        and a corresponding reconstructed/decoded enhancement layer        picture may be used for predicting a residual block of the        current enhancement layer block. A residual block may be added        for example to a motion-compensated prediction block to obtain a        final prediction block for the current enhancement layer block.    -   Intra prediction, where pixel or sample values can be predicted        by spatial mechanisms which involve finding and indicating a        spatial region relationship.

In the syntax prediction, which may also be referred to as parameterprediction, syntax elements and/or syntax element values and/orvariables derived from syntax elements are predicted from syntaxelements (de)coded earlier and/or variables derived earlier.Non-limiting examples of syntax prediction are provided below:

-   -   In motion vector prediction, motion vectors e.g. for inter        and/or inter-view prediction may be coded differentially with        respect to a block-specific predicted motion vector. In many        video codecs, the predicted motion vectors are created in a        predefined way, for example by calculating the median of the        encoded or decoded motion vectors of the adjacent blocks.        Another way to create motion vector predictions, sometimes        referred to as advanced motion vector prediction (AMVP), is to        generate a list of candidate predictions from adjacent blocks        and/or co-located blocks in temporal reference pictures and        signalling the chosen candidate as the motion vector predictor.        In addition to predicting the motion vector values, the        reference index of previously coded/decoded picture can be        predicted. The reference index is typically predicted from        adjacent blocks and/or co-located blocks in temporal reference        picture. Differential coding of motion vectors is typically        disabled across slice boundaries.    -   The block partitioning, e.g. from CTU to CUs and down to PUs,        may be predicted.    -   In filter parameter prediction, the filtering parameters e.g.        for sample adaptive offset may be predicted.

Prediction approaches using image information from a previously codedimage can also be called as inter prediction methods which may also bereferred to as temporal prediction and motion compensation. Predictionapproaches using image information within the same image can also becalled as intra prediction methods.

The second phase is one of coding the error between the predicted blockof pixels or samples and the original block of pixels or samples. Thismay be accomplished by transforming the difference in pixel or samplevalues using a specified transform. This transform may be e.g. aDiscrete Cosine Transform (DCT) or a variant thereof. After transformingthe difference, the transformed difference is quantized and entropycoded.

By varying the fidelity of the quantization process, the encoder cancontrol the balance between the accuracy of the pixel or samplerepresentation (i.e. the visual quality of the picture) and the size ofthe resulting encoded video representation (i.e. the file size ortransmission bit rate).

The decoder reconstructs the output video by applying a predictionmechanism similar to that used by the encoder in order to form apredicted representation of the pixel or sample blocks (using the motionor spatial information created by the encoder and included in thecompressed representation of the image) and prediction error decoding(the inverse operation of the prediction error coding to recover thequantized prediction error signal in the spatial domain).

After applying pixel or sample prediction and error decoding processesthe decoder combines the prediction and the prediction error signals(the pixel or sample values) to form the output video frame.

The decoder (and encoder) may also apply additional filtering processesin order to improve the quality of the output video before passing itfor display and/or storing as a prediction reference for the forthcomingpictures in the video sequence.

In many video codecs, including H.264/AVC and HEVC, motion informationis indicated by motion vectors associated with each motion compensatedimage block. Each of these motion vectors represents the displacement ofthe image block in the picture to be coded (in the encoder) or decoded(at the decoder) and the prediction source block in one of thepreviously coded or decoded images (or picture). H.264/AVC and HEVC, asmany other video compression standards, divide a picture into a mesh ofrectangles, for each of which a similar block in one of the referencepictures is indicated for inter prediction. The location of theprediction block is coded as a motion vector that indicates the positionof the prediction block relative to the block being coded.

H.264/AVC and HEVC include a concept of picture order count (POC). Avalue of POC is derived for each picture and is non-decreasing withincreasing picture position in output order. POC therefore indicates theoutput order of pictures. POC may be used in the decoding process forexample for implicit scaling of motion vectors in the temporal directmode of bi-predictive slices, for implicitly derived weights in weightedprediction, and for reference picture list initialization. Furthermore,POC may be used in the verification of output order conformance. InH.264/AVC, POC is specified relative to the previous IDR picture or apicture containing a memory management control operation marking allpictures as “unused for reference”.

Inter prediction process may be characterized using one or more of thefollowing factors.

The accuracy of motion vector representation. For example, motionvectors may be of quarter-pixel accuracy, and sample values infractional-pixel positions may be obtained using a finite impulseresponse (FIR) filter.

Block partitioning for inter prediction. Many coding standards,including H.264/AVC and HEVC, allow selection of the size and shape ofthe block for which a motion vector is applied for motion-compensatedprediction in the encoder, and indicating the selected size and shape inthe bitstream so that decoders can reproduce the motion-compensatedprediction done in the encoder.

Number of reference pictures for inter prediction. The sources of interprediction are previously decoded pictures. Many coding standards,including H.264/AVC and HEVC, enable storage of multiple referencepictures for inter prediction and selection of the used referencepicture on a block basis. For example, reference pictures may beselected on macroblock or macroblock partition basis in H.264/AVC and onPU or CU basis in HEVC. Many coding standards, such as H.264/AVC andHEVC, include syntax structures in the bitstream that enable decoders tocreate one or more reference picture lists. A reference picture index toa reference picture list may be used to indicate which one of themultiple reference pictures is used for inter prediction for aparticular block. A reference picture index may be coded by an encoderinto the bitstream in some inter coding modes or it may be derived (byan encoder and a decoder) for example using neighboring blocks in someother inter coding modes.

Motion vector prediction. In order to represent motion vectorsefficiently in bitstreams, motion vectors may be coded differentiallywith respect to a block-specific predicted motion vector. In many videocodecs, the predicted motion vectors are created in a predefined way,for example by calculating the median of the encoded or decoded motionvectors of the adjacent blocks. Another way to create motion vectorpredictions, sometimes referred to as advanced motion vector prediction(AMVP), is to generate a list of candidate predictions from adjacentblocks and/or co-located blocks in temporal reference pictures andsignalling the chosen candidate as the motion vector predictor. Inaddition to predicting the motion vector values, the reference index ofpreviously coded/decoded picture can be predicted. The reference indexmay be predicted e.g. from adjacent blocks and/or co-located blocks intemporal reference picture. Differential coding of motion vectors may bedisabled across slice boundaries.

Multi-hypothesis motion-compensated prediction. H.264/AVC and HEVCenable the use of a single prediction block in P slices (herein referredto as uni-predictive slices) or a linear combination of twomotion-compensated prediction blocks for bi-predictive slices, which arealso referred to as B slices. Individual blocks in B slices may bebi-predicted, uni-predicted, or intra-predicted, and individual blocksin P slices may be uni-predicted or intra-predicted. The referencepictures for a bi-predictive picture may not be limited to be thesubsequent picture and the previous picture in output order, but ratherany reference pictures may be used. In many coding standards, such asH.264/AVC and HEVC, one reference picture list, referred to as referencepicture list 0, is constructed for P slices, and two reference picturelists, list 0 and list 1, are constructed for B slices. For B slices,when prediction in forward direction may refer to prediction from areference picture in reference picture list 0, and prediction inbackward direction may refer to prediction from a reference picture inreference picture list 1, even though the reference pictures forprediction may have any decoding or output order relation to each otheror to the current picture.

Weighted prediction. Many coding standards use a prediction weight of 1for prediction blocks of inter (P) pictures and 0.5 for each predictionblock of a B picture (resulting into averaging). H.264/AVC allowsweighted prediction for both P and B slices. In implicit weightedprediction, the weights are proportional to picture order counts (POC),while in explicit weighted prediction, prediction weights are explicitlyindicated.

In many video codecs, the prediction residual after motion compensationis first transformed with a transform kernel (like DCT) and then coded.The reason for this is that often there still exists some correlationamong the residual and transform can in many cases help reduce thiscorrelation and provide more efficient coding.

In a draft HEVC, each PU has prediction information associated with itdefining what kind of a prediction is to be applied for the pixelswithin that PU (e.g. motion vector information for inter predicted PUsand intra prediction directionality information for intra predictedPUs). Similarly each TU is associated with information describing theprediction error decoding process for the samples within the TU(including e.g. DCT coefficient information). It may be signaled at CUlevel whether prediction error coding is applied or not for each CU. Inthe case there is no prediction error residual associated with the CU,it can be considered there are no TUs for the CU.

In some coding formats and codecs, a distinction is made betweenso-called short-term and long-term reference pictures. This distinctionmay affect some decoding processes such as motion vector scaling in thetemporal direct mode or implicit weighted prediction. If both of thereference pictures used for the temporal direct mode are short-termreference pictures, the motion vector used in the prediction may bescaled according to the picture order count difference between thecurrent picture and each of the reference pictures. However, if at leastone reference picture for the temporal direct mode is a long-termreference picture, default scaling of the motion vector may be used, forexample scaling the motion to half may be used. Similarly, if ashort-term reference picture is used for implicit weighted prediction,the prediction weight may be scaled according to the POC differencebetween the POC of the current picture and the POC of the referencepicture. However, if a long-term reference picture is used for implicitweighted prediction, a default prediction weight may be used, such as0.5 in implicit weighted prediction for bi-predicted blocks.

Some video coding formats, such as H.264/AVC, include the frame_numsyntax element, which is used for various decoding processes related tomultiple reference pictures. In H.264/AVC, the value of frame_num forIDR pictures is 0. The value of frame_num for non-IDR pictures is equalto the frame_num of the previous reference picture in decoding orderincremented by 1 (in modulo arithmetic, i.e., the value of frame_numwrap over to 0 after a maximum value of frame_num).

A syntax structure for (decoded) reference picture marking may exist ina video coding system. For example, when the decoding of the picture hasbeen completed, the decoded reference picture marking syntax structure,if present, may be used to adaptively mark pictures as “unused forreference” or “used for long-term reference”. If the decoded referencepicture marking syntax structure is not present and the number ofpictures marked as “used for reference” can no longer increase, asliding window reference picture marking may be used, which basicallymarks the earliest (in decoding order) decoded reference picture asunused for reference.

H.264/AVC specifies the process for decoded reference picture marking inorder to control the memory consumption in the decoder. The maximumnumber of reference pictures used for inter prediction, referred to asM, is determined in the sequence parameter set. When a reference pictureis decoded, it is marked as “used for reference”. If the decoding of thereference picture caused more than M pictures marked as “used forreference”, at least one picture is marked as “unused for reference”.There are two types of operation for decoded reference picture marking:adaptive memory control and sliding window. The operation mode fordecoded reference picture marking is selected on picture basis. Theadaptive memory control enables explicit signaling which pictures aremarked as “unused for reference” and may also assign long-term indicesto short-term reference pictures. The adaptive memory control mayrequire the presence of memory management control operation (MMCO)parameters in the bitstream. MMCO parameters may be included in adecoded reference picture marking syntax structure. If the slidingwindow operation mode is in use and there are M pictures marked as “usedfor reference”, the short-term reference picture that was the firstdecoded picture among those short-term reference pictures that aremarked as “used for reference” is marked as “unused for reference”. Inother words, the sliding window operation mode results intofirst-in-first-out buffering operation among short-term referencepictures.

One of the memory management control operations in H.264/AVC causes allreference pictures except for the current picture to be marked as“unused for reference”. An instantaneous decoding refresh (IDR) picturecontains only intra-coded slices and causes a similar “reset” ofreference pictures.

In a draft HEVC, reference picture marking syntax structures and relateddecoding processes have been replaced with a reference picture set (RPS)syntax structure and decoding process are used instead for a similarpurpose. A reference picture set valid or active for a picture includesall the reference pictures used as reference for the picture and all thereference pictures that are kept marked as “used for reference” for anysubsequent pictures in decoding order. There are six subsets of thereference picture set, which are referred to as RefPicSetStCurr0,RefPicSetStCurr1, RefPicSetStFoll0, RefPicSetStFoll1, RefPicSetLtCurr,and RefPicSetLtFoll. The notation of the six subsets is as follows.“Curr” refers to the reference pictures that are included in thereference picture lists of the current picture and hence may be used asinter prediction reference for the current picture. “Foll” refers toreference pictures that are not included in the reference picture listsof the current picture but may be used in subsequent pictures indecoding order as reference pictures. “St” refers to short-termreference pictures, which may generally be identified through a certainnumber of least significant bits of their POC value. “Lt” refers tolong-term reference pictures, which are specifically identified andgenerally have a greater difference of POC values relative to thecurrent picture than what can be represented by the mentioned certainnumber of least significant bits. “0” refers to those reference picturesthat have a smaller POC value than that of the current picture. “1”refers to those reference pictures that have a greater POC value thanthat of the current picture. RefPicSetStCurr0, RefPicSetStCurr1,RefPicSetStFoll0 and RefPicSetStFoll1 are collectively referred to asthe short-term subset of the reference picture set. RefPicSetLtCurr andRefPicSetLtFoll are collectively referred to as the long-term subset ofthe reference picture set.

In HEVC, a reference picture set may be specified in a picture parameterset and taken into use in the slice header through an index to thereference picture set. A reference picture set may also be specified ina slice header. A long-term subset of a reference picture set isgenerally specified only in a slice header, while the short-term subsetsof the same reference picture set may be specified in the pictureparameter set or slice header. A reference picture set may be codedindependently or may be predicted from another reference picture set(known as inter-RPS prediction). When a reference picture set isindependently coded, the syntax structure includes up to three loopsiterating over different types of reference pictures; short-termreference pictures with lower POC value than the current picture,short-term reference pictures with higher POC value than the currentpicture, and long-term reference pictures. Each loop entry specifies apicture to be marked as “used for reference”. In general, the picture isspecified with a differential POC value. The inter-RPS predictionexploits the fact that the reference picture set of the current picturecan be predicted from the reference picture set of a previously decodedpicture. This is because all the reference pictures of the currentpicture are either reference pictures of the previous picture or thepreviously decoded picture itself. It is only necessary to indicatewhich of these pictures should be reference pictures and be used for theprediction of the current picture. In both types of reference pictureset coding, a flag (used_by_curr_pic_X_flag) is additionally sent foreach reference picture indicating whether the reference picture is usedfor reference by the current picture (included in a *Curr list) or not(included in a *Foll list). Pictures that are included in the referencepicture set used by the current slice are marked as “used forreference”, and pictures that are not in the reference picture set usedby the current slice are marked as “unused for reference”. If thecurrent picture is an IDR picture, RefPicSetStCurr0, RefPicSetStCurr1,RefPicSetStFoll0, RefPicSetStFoll1, RefPicSetLtCurr, and RefPicSetLtFollare all set to empty.

A Decoded Picture Buffer (DPB) may be used in the encoder and/or in thedecoder. There are two reasons to buffer decoded pictures, forreferences in inter prediction and for reordering decoded pictures intooutput order. As H.264/AVC and HEVC provide a great deal of flexibilityfor both reference picture marking and output reordering, separatebuffers for reference picture buffering and output picture buffering maywaste memory resources. Hence, the DPB may include a unified decodedpicture buffering process for reference pictures and output reordering.A decoded picture may be removed from the DPB when it is no longer usedas a reference and is not needed for output.

In many coding modes of H.264/AVC and HEVC, the reference picture forinter prediction is indicated with an index to a reference picture list.The index may be coded with CABAC or variable length coding. In general,the smaller the index is, the shorter the corresponding syntax elementmay become. In H.264/AVC and HEVC, two reference picture lists(reference picture list 0 and reference picture list 1) are generatedfor each bi-predictive (B) slice, and one reference picture list(reference picture list 0) is formed for each inter-coded (P) slice. Inaddition, for a B slice in a draft HEVC standard, a combined list (ListC) may be constructed after the final reference picture lists (List 0and List 1) have been constructed. The combined list may be used foruni-prediction (also known as uni-directional prediction) within Bslices. However, in the final H.265/HEVC standard, no combined list isconstructed.

A reference picture list, such as the reference picture list 0 and thereference picture list 1, may be constructed in two steps: First, aninitial reference picture list is generated. The initial referencepicture list may be generated for example on the basis of frame_num,POC, temporal_id, or information on the prediction hierarchy such as aGOP structure, or any combination thereof. Second, the initial referencepicture list may be reordered by reference picture list reordering(RPLR) commands, also known as reference picture list modificationsyntax structure, which may be contained in slice headers. The RPLRcommands indicate the pictures that are ordered to the beginning of therespective reference picture list. This second step may also be referredto as the reference picture list modification process, and the RPLRcommands may be included in a reference picture list modification syntaxstructure. If reference picture sets are used, the reference picturelist 0 may be initialized to contain RefPicSetStCurr0 first, followed byRefPicSetStCurr1, followed by RefPicSetLtCurr. Reference picture list 1may be initialized to contain RefPicSetStCurr1 first, followed byRefPicSetStCurr0. The initial reference picture lists may be modifiedthrough the reference picture list modification syntax structure, wherepictures in the initial reference picture lists may be identifiedthrough an entry index to the list.

Since multiview video provides encoders and decoders the possibility toutilize inter-view redundancy, decoded inter-view frames may be includedin the reference picture list(s) as well.

Examples of motion vector prediction schemes and related coding modesare provided in the next paragraphs.

In addition to the motion-compensated macroblock modes for which adifferential motion vector is coded, a P macroblock may also be coded inthe so-called P_Skip type in H.264/AVC. For this coding type, nodifferential motion vector, reference index, or quantized predictionerror signal is coded into the bitstream. The reference picture of amacroblock coded with the P_Skip type has index 0 in reference picturelist 0. The motion vector used for reconstructing the P_Skip macroblockis obtained using median motion vector prediction for the macroblockwithout any differential motion vector being added. P_Skip may bebeneficial for compression efficiency particularly in areas where themotion field is smooth.

In B slices of H.264/AVC, four different types of inter prediction aresupported: uni-predictive from reference picture list 0, uni-directionalfrom reference picture list 1, bi-predictive, direct prediction, andB_skip. The type of inter prediction can be selected separately for eachmacroblock partition. B slices utilize a similar macroblock partitioningas P slices. For a bi-predictive macroblock partition, the predictionsignal is formed by a weighted average of motion-compensated list 0 andlist 1 prediction signals. Reference indices, motion vector differences,as well as quantized prediction error signal may be coded foruni-predictive and bi-predictive B macroblock partitions.

Two direct modes are included in H.264/AVC, temporal direct and spatialdirect, and one of them can be selected into use for a slice in a sliceheader, although their use may be constrained further for example inprofiles or alike. In the temporal direct mode, the reference index forreference picture list 1 is set to 0 and the reference index forreference picture list 0 is set to point to the reference picture thatis used in the co-located block (compared to the current block cb) ofthe reference picture having index 0 in the reference picture list 1 ifthat reference picture is available, or set to 0 if that referencepicture is not available. The motion vector predictor for cb isessentially derived by considering the motion information within aco-located block of the reference picture having index 0 in referencepicture list 1. Motion vector predictors for a temporal direct block arederived by scaling a motion vector from the co-located block. Thescaling is proportional to picture order count differences between thecurrent picture and the reference pictures associated with the inferredreference indexes in list 0 and list 1, and by selecting the sign forthe motion vector predictor depending on which reference picture list itis using.

In spatial direct mode of H.264/AVC, motion information of spatiallyadjacent blocks is exploited. Motion vector prediction in spatial directmode can be divided into three steps: reference index determination,determination of uni- or bi-prediction, and motion vector prediction. Inthe first step, the reference picture with the minimum non-negativereference index (i.e., non-intra block) is selected from each ofreference picture list 0 and reference picture list 1 of the neighboringblocks A, B, and C (where A is the adjacent block on the left of thecurrent block, B is the adjacent block above the current block and C isthe adjacent block on the top-right side of the current block). If nonon-negative reference index exists in reference picture list 0 of theneighboring blocks A, B, and C, and likewise no non-negative referenceindex exists in reference picture list 1 of the neighboring blocks A, B,and C, reference index 0 is selected for both reference picture lists.

The use of uni- or bi-prediction for H.264/AVC spatial direct mode isdetermined as follows: If a minimum non-negative reference index forboth reference picture lists was found in the reference indexdetermination step, bi-prediction is used. If a minimum non-negativereference index for either but not both of reference picture list 0 orreference picture list 1 was found in the reference index determinationstep, uni-prediction from either reference picture list 0 or referencepicture list 1, respectively, is used.

In the motion vector prediction for H.264/AVC spatial direct mode,certain conditions, such as whether a negative reference index wasconcluded in the first step, are checked and, if fulfilled, a zeromotion vector is determined. Otherwise, the motion vector predictor isderived similarly to the motion vector predictor of P blocks using themotion vectors of spatially adjacent blocks A, B, and C.

No motion vector differences or reference indices are present in thebitstream for a direct mode block in H.264/AVC, while quantizedprediction error signal may be coded and present therefore present inthe bitstream. A B_skip macroblock mode in H.264/AVC is similar to thedirect mode but no prediction error signal is coded and included in thebitstream.

H.265/HEVC includes two motion vector prediction schemes, namely theadvanced motion vector prediction (AMVP) and the merge mode. In the AMVPor the merge mode, a list of motion vector candidates is derived for aPU. There are two kinds of candidates: spatial candidates and temporalcandidates, where temporal candidates may also be referred to as TMVPcandidates. The sources of the candidate motion vector predictors arepresented in FIG. 5. X stands for the current prediction unit. A₀, A₁,B₀, B₁, B₂ in FIG. 5a are spatial candidates while C₀, C₁ in FIG. 5b aretemporal candidates.

A candidate list derivation may be performed for example as follows,while it should be understood that other possibilities exist forcandidate list derivation. If the occupancy of the candidate list is notat maximum, the spatial candidates are included in the candidate listfirst if they are available and not already exist in the candidate list.After that, if occupancy of the candidate list is not yet at maximum, atemporal candidate is included in the candidate list. If the number ofcandidates still does not reach the maximum allowed number, the combinedbi-predictive candidates (for B slices) and a zero motion vector areadded in. After the candidate list has been constructed, the encoderdecides the final motion information from candidates for example basedon a rate-distortion optimization (RDO) decision and encodes the indexof the selected candidate into the bitstream. Likewise, the decoderdecodes the index of the selected candidate from the bitstream,constructs the candidate list, and uses the decoded index to select amotion vector predictor from the candidate list.

In H.265/HEVC, AMVP and the merge mode may be characterized as follows.In AMVP, the encoder indicates whether uni-prediction or bi-predictionis used and which reference pictures are used as well as encodes amotion vector difference. In the merge mode, only the chosen candidatefrom the candidate list is encoded into the bitstream indicating thecurrent prediction unit has the same motion information as that of theindicated predictor. Thus, the merge mode creates regions composed ofneighboring prediction blocks sharing identical motion information,which is only signaled once for each region. Another difference betweenAMVP and the merge mode in H.265/HEVC is that the maximum number ofcandidates of AMVP is 2 while that of the merge mode is 5.

The advanced motion vector prediction may operate for example asfollows, while other similar realizations of advanced motion vectorprediction are also possible for example with different candidateposition sets and candidate locations with candidate position sets. Twospatial motion vector predictors (MVPs) may be derived and a temporalmotion vector predictor (TMVP) may be derived. They may be selectedamong the positions: three spatial motion vector predictor candidatepositions located above the current prediction block (B0, B1, B2) andtwo on the left (A0, A1). The first motion vector predictor that isavailable (e.g. resides in the same slice, is inter-coded, etc.) in apre-defined order of each candidate position set, (B0, B1, B2) or (A0,A1), may be selected to represent that prediction direction (up or left)in the motion vector competition. A reference index for the temporalmotion vector predictor may be indicated by the encoder in the sliceheader (e.g. as a collocated_ref_idx syntax element). The motion vectorobtained from the co-located picture may be scaled according to theproportions of the picture order count differences of the referencepicture of the temporal motion vector predictor, the co-located picture,and the current picture. Moreover, a redundancy check may be performedamong the candidates to remove identical candidates, which can lead tothe inclusion of a zero motion vector in the candidate list. The motionvector predictor may be indicated in the bitstream for example byindicating the direction of the spatial motion vector predictor (up orleft) or the selection of the temporal motion vector predictorcandidate.

The merging/merge mode/process/mechanism may operate for example asfollows, while other similar realizations of the merge mode are alsopossible for example with different candidate position sets andcandidate locations with candidate position sets.

In the merging/merge mode/process/mechanism, where all the motioninformation of a block/PU is predicted and used without anymodification/correction. The aforementioned motion information for a PUmay comprise one or more of the following: 1) The information whether‘the PU is uni-predicted using only reference picture list0’ or ‘the PUis uni-predicted using only reference picture list1’ or ‘the PU isbi-predicted using both reference picture list0 and list1’; 2) Motionvector value corresponding to the reference picture list0, which maycomprise a horizontal and vertical motion vector component; 3) Referencepicture index in the reference picture list0 and/or an identifier of areference picture pointed to by the Motion vector corresponding toreference picture list 0, where the identifier of a reference picturemay be for example a picture order count value, a layer identifier value(for inter-layer prediction), or a pair of a picture order count valueand a layer identifier value; 4) Information of the reference picturemarking of the reference picture, e.g. information whether the referencepicture was marked as “used for short-term reference” or “used forlong-term reference”; 5)-7) The same as 2)-4), respectively, but forreference picture list1.

Similarly, predicting the motion information is carried out using themotion information of adjacent blocks and/or co-located blocks intemporal reference pictures. A list, often called as a merge list, maybe constructed by including motion prediction candidates associated withavailable adjacent/co-located blocks and the index of selected motionprediction candidate in the list is signalled and the motion informationof the selected candidate is copied to the motion information of thecurrent PU. When the merge mechanism is employed for a whole CU and theprediction signal for the CU is used as the reconstruction signal, i.e.prediction residual is not processed, this type of coding/decoding theCU is typically named as skip mode or merge based skip mode. In additionto the skip mode, the merge mechanism may also be employed forindividual PUs (not necessarily the whole CU as in skip mode) and inthis case, prediction residual may be utilized to improve predictionquality. This type of prediction mode is typically named as aninter-merge mode.

One of the candidates in the merge list and/or the candidate list forAMVP or any similar motion vector candidate list may be a TMVP candidateor alike, which may be derived from the collocated block within anindicated or inferred reference picture, such as the reference pictureindicated for example in the slice header. In HEVC, the referencepicture list to be used for obtaining a collocated partition is chosenaccording to the collocated_from_10_flag syntax element in the sliceheader. When the flag is equal to 1, it specifies that the picture thatcontains the collocated partition is derived from list 0, otherwise thepicture is derived from list 1. When collocated_from_10_flag is notpresent, it is inferred to be equal to 1. The collocated_ref_idx in theslice header specifies the reference index of the picture that containsthe collocated partition. When the current slice is a P slice,collocated_ref_idx refers to a picture in list 0. When the current sliceis a B slice, collocated_ref_idx refers to a picture in list 0 ifcollocated_from_10 is 1, otherwise it refers to a picture in list 1.collocated_ref_idx always refers to a valid list entry, and theresulting picture is the same for all slices of a coded picture. Whencollocated_ref_idx is not present, it is inferred to be equal to 0.

In HEVC the so-called target reference index for temporal motion vectorprediction in the merge list is set as 0 when the motion coding mode isthe merge mode. When the motion coding mode in HEVC utilizing thetemporal motion vector prediction is the advanced motion vectorprediction mode, the target reference index values are explicitlyindicated (e.g. per each PU).

In HEVC, the availability of a candidate predicted motion vector (PMV)may be determined as follows (both for spatial and temporal candidates)(SRTP=short-term reference picture, LRTP=long-term reference picture):

reference picture for target reference picture for candidate PMVreference index candidate PMV availability STRP STRP “available” (andscaled) STRP LTRP “unavailable” LTRP STRP “unavailable” LTRP LTRP“available” (but not scaled)

In HEVC, when the target reference index value has been determined, themotion vector value of the temporal motion vector prediction may bederived as follows: The motion vector PMV at the block that iscollocated with the bottom-right neighbor (location C₀ in FIG. 5b ) ofthe current prediction unit is obtained. The picture where thecollocated block resides may be e.g. determined according to thesignalled reference index in the slice header as described above. If thePMV at location C₀ is not available, the motion vector PMV at locationC₁ (see FIG. 5b ) of the collocated picture is obtained. The determinedavailable motion vector PMV at the co-located block is scaled withrespect to the ratio of a first picture order count difference and asecond picture order count difference. The first picture order countdifference is derived between the picture containing the co-locatedblock and the reference picture of the motion vector of the co-locatedblock. The second picture order count difference is derived between thecurrent picture and the target reference picture. If one but not both ofthe target reference picture and the reference picture of the motionvector of the collocated block is a long-term reference picture (whilethe other is a short-term reference picture), the TMVP candidate may beconsidered unavailable. If both of the target reference picture and thereference picture of the motion vector of the collocated block arelong-term reference pictures, no POC-based motion vector scaling may beapplied.

Motion parameter types or motion information may include but are notlimited to one or more of the following types:

-   -   an indication of a prediction type (e.g. intra prediction,        uni-prediction, bi-prediction) and/or a number of reference        pictures;    -   an indication of a prediction direction, such as inter (a.k.a.        temporal) prediction, inter-layer prediction, inter-view        prediction, view synthesis prediction (VSP), and inter-component        prediction (which may be indicated per reference picture and/or        per prediction type and where in some embodiments inter-view and        view-synthesis prediction may be jointly considered as one        prediction direction) and/or an indication of a reference        picture type, such as a short-term reference picture and/or a        long-term reference picture and/or an inter-layer reference        picture (which may be indicated e.g. per reference picture)    -   a reference index to a reference picture list and/or any other        identifier of a reference picture (which may be indicated e.g.        per reference picture and the type of which may depend on the        prediction direction and/or the reference picture type and which        may be accompanied by other relevant pieces of information, such        as the reference picture list or alike to which reference index        applies);    -   a horizontal motion vector component (which may be indicated        e.g. per prediction block or per reference index or alike);    -   a vertical motion vector component (which may be indicated e.g.        per prediction block or per reference index or alike);    -   one or more parameters, such as picture order count difference        and/or a relative camera separation between the picture        containing or associated with the motion parameters and its        reference picture, which may be used for scaling of the        horizontal motion vector component and/or the vertical motion        vector component in one or more motion vector prediction        processes (where said one or more parameters may be indicated        e.g. per each reference picture or each reference index or        alike);    -   coordinates of a block to which the motion parameters and/or        motion information applies, e.g. coordinates of the top-left        sample of the block in luma sample units;    -   extents (e.g. a width and a height) of a block to which the        motion parameters and/or motion information applies.

In general, motion vector prediction mechanisms, such as those motionvector prediction mechanisms presented above as examples, may includeprediction or inheritance of certain pre-defined or indicated motionparameters.

A motion field associated with a picture may be considered to compriseof a set of motion information produced for every coded block of thepicture. A motion field may be accessible by coordinates of a block, forexample. A motion field may be used for example in TMVP or any othermotion prediction mechanism where a source or a reference for predictionother than the current (de)coded picture is used.

Different spatial granularity or units may be applied to representand/or store a motion field. For example, a regular grid of spatialunits may be used. For example, a picture may be divided intorectangular blocks of certain size (with the possible exception ofblocks at the edges of the picture, such as on the right edge and thebottom edge). For example, the size of the spatial unit may be equal tothe smallest size for which a distinct motion can be indicated by theencoder in the bitstream, such as a 4×4 block in luma sample units. Forexample, a so-called compressed motion field may be used, where thespatial unit may be equal to a pre-defined or indicated size, such as a16×16 block in luma sample units, which size may be greater than thesmallest size for indicating distinct motion. For example, an HEVCencoder and/or decoder may be implemented in a manner that a motion datastorage reduction (MDSR) or motion field compression is performed foreach decoded motion field (prior to using the motion field for anyprediction between pictures). In an HEVC implementation, MDSR may reducethe granularity of motion data to 16×16 blocks in luma sample units bykeeping the motion applicable to the top-left sample of the 16×16 blockin the compressed motion field. The encoder may encode indication(s)related to the spatial unit of the compressed motion field as one ormore syntax elements and/or syntax element values for example in asequence-level syntax structure, such as a video parameter set or asequence parameter set. In some (de)coding methods and/or devices, amotion field may be represented and/or stored according to the blockpartitioning of the motion prediction (e.g. according to predictionunits of the HEVC standard). In some (de)coding methods and/or devices,a combination of a regular grid and block partitioning may be applied sothat motion associated with partitions greater than a pre-defined orindicated spatial unit size is represented and/or stored associated withthose partitions, whereas motion associated with partitions smaller thanor unaligned with a pre-defined or indicated spatial unit size or gridis represented and/or stored for the pre-defined or indicated units.

Many video coding standards specify buffering models and bufferingparameters for bitstreams. Such buffering models may be calledHypothetical Reference Decoder (HRD) or Video Buffer Verifier (VBV). Astandard compliant bitstream complies with the buffering model with aset of buffering parameters specified in the corresponding standard.Such buffering parameters for a bitstream may be explicitly orimplicitly signaled. ‘Implicitly signaled’ means for example that thedefault buffering parameter values according to the profile and levelapply. The HRD/VBV parameters are used, among other things, to imposeconstraints on the bit rate variations of compliant bitstreams.

HRD conformance checking may concern for example the following two typesof bitstreams: The first such type of bitstream, called Type Ibitstream, is a NAL unit stream containing only the VCL NAL units andfiller data NAL units for all access units in the bitstream. The secondtype of bitstream, called a Type II bitstream, may contain, in additionto the VCL NAL units and filler data NAL units for all access units inthe bitstream, additional non-VCL NAL units other than filler data NALunits and/or syntax elements such as leading_zero_8 bits, zero byte,start_code_prefix_one_3 bytes, and trailing_zero_8 bits that form a bytestream from the NAL unit stream.

Two types of HRD parameters (NAL HRD parameters and VCL HRD parameters)may be used. The HRD parameter may be indicated through video usabilityinformation included in the sequence parameter set syntax structure.

Buffering and picture timing parameters (e.g. included in sequenceparameter sets and picture parameter sets referred to in the VCL NALunits and in buffering period and picture timing SEI messages) may beconveyed to the HRD, in a timely manner, either in the bitstream (bynon-VCL NAL units), or by out-of-band means externally from thebitstream e.g. using a signalling mechanism, such as media parametersincluded in the media line of a session description formatted e.g.according to the Session Description Protocol (SDP). For the purpose ofcounting bits in the HRD, only the appropriate bits that are actuallypresent in the bitstream may be counted. When the content of a non-VCLNAL unit is conveyed for the application by some means other thanpresence within the bitstream, the representation of the content of thenon-VCL NAL unit may or may not use the same syntax as would be used ifthe non-VCL NAL unit were in the bitstream.

The HRD may contain a coded picture buffer (CPB), an instantaneousdecoding process, a decoded picture buffer (DPB), and output cropping.

The CPB may operate on decoding unit basis. A decoding unit may be anaccess unit or it may be a subset of an access unit, such as an integernumber of NAL units. The selection of the decoding unit may be indicatedby an encoder in the bitstream.

The HRD may operate as follows. Data associated with decoding units thatflow into the CPB according to a specified arrival schedule may bedelivered by the Hypothetical Stream Scheduler (HSS). The arrivalschedule may be determined by the encoder and indicated for examplethrough picture timing SEI messages, and/or the arrival schedule may bederived for example based on a bitrate which may be indicated forexample as part of HRD parameters in video usability information (whichmay be included in the sequence parameter set). The HRD parameters invideo usability information may contain many sets of parameters, eachfor different bitrate or delivery schedule. The data associated witheach decoding unit may be removed and decoded instantaneously by theinstantaneous decoding process at CPB removal times. A CPB removal timemay be determined for example using an initial CPB buffering delay,which may be determined by the encoder and indicated for example througha buffering period SEI message, and differential removal delaysindicated for each picture for example though picture timing SEImessages. Each decoded picture is placed in the DPB. A decoded picturemay be removed from the DPB at the later of the DPB output time or thetime that it becomes no longer needed for inter-prediction reference.Thus, the operation of the CPB of the HRD may comprise timing ofbitstream arrival, timing of decoding unit removal and decoding ofdecoding unit, whereas the operation of the DPB of the HRD may compriseremoval of pictures from the DPB, picture output, and current decodedpicture marking and storage.

The HRD may be used to check conformance of bitstreams and decoders.

Bitstream conformance requirements of the HRD may comprise for examplethe following and/or alike. The CPB is required not to overflow(relative to the size which may be indicated for example within HRDparameters of video usability information) or underflow (i.e. theremoval time of a decoding unit cannot be smaller than the arrival timeof the last bit of that decoding unit). The number of pictures in theDPB may be required to be smaller than or equal to a certain maximumnumber, which may be indicated for example in the sequence parameterset. All pictures used as prediction references may be required to bepresent in the DPB. It may be required that the interval for outputtingconsecutive pictures from the DPB is not smaller than a certain minimum.

Decoder conformance requirements of the HRD may comprise for example thefollowing and/or alike. A decoder claiming conformance to a specificprofile and level may be required to decode successfully all conformingbitstreams specified for decoder conformance provided that all sequenceparameter sets and picture parameter sets referred to in the VCL NALunits, and appropriate buffering period and picture timing SEI messagesare conveyed to the decoder, in a timely manner, either in the bitstream(by non-VCL NAL units), or by external means. There may be two types ofconformance that can be claimed by a decoder: output timing conformanceand output order conformance.

To check conformance of a decoder, test bitstreams conforming to theclaimed profile and level may be delivered by a hypothetical streamscheduler (HSS) both to the HRD and to the decoder under test (DUT). Allpictures output by the HRD may also be required to be output by the DUTand, for each picture output by the HRD, the values of all samples thatare output by the DUT for the corresponding picture may also be requiredto be equal to the values of the samples output by the HRD.

For output timing decoder conformance, the HSS may operate e.g. withdelivery schedules selected from those indicated in the HRD parametersof video usability information, or with “interpolated” deliveryschedules. The same delivery schedule may be used for both the HRD andDUT. For output timing decoder conformance, the timing (relative to thedelivery time of the first bit) of picture output may be required to bethe same for both HRD and the DUT up to a fixed delay.

For output order decoder conformance, the HSS may deliver the bitstreamto the DUT “by demand” from the DUT, meaning that the HSS delivers bits(in decoding order) only when the DUT requires more bits to proceed withits processing. The HSS may deliver the bitstream to the HRD by one ofthe schedules specified in the bitstream such that the bit rate and CPBsize are restricted. The order of pictures output may be required to bethe same for both HRD and the DUT.

In scalable video coding, a video signal can be encoded into a baselayer and one or more enhancement layers. An enhancement layer mayenhance the temporal resolution (i.e., the frame rate), the spatialresolution, or simply the quality of the video content represented byanother layer or part thereof. Each layer together with all itsdependent layers is one representation of the video signal at a certainspatial resolution, temporal resolution and quality level. In thisdocument, we refer to a scalable layer together with all of itsdependent layers as a “scalable layer representation”. The portion of ascalable bitstream corresponding to a scalable layer representation canbe extracted and decoded to produce a representation of the originalsignal at certain fidelity.

SVC uses an inter-layer prediction mechanism, wherein certaininformation can be predicted from layers other than the currentlyreconstructed layer or the next lower layer. Information that could beinter-layer predicted includes intra texture, motion and residual data.Inter-layer motion prediction includes the prediction of block codingmode, header information, etc., wherein motion from the lower layer maybe used for prediction of the higher layer. In case of intra coding, aprediction from surrounding macroblocks or from co-located macroblocksof lower layers is possible. These prediction techniques do not employinformation from earlier coded access units and hence, are referred toas intra prediction techniques. Furthermore, residual data from lowerlayers can also be employed for prediction of the current layer.

As indicated earlier, MVC is an extension of H.264/AVC. Many of thedefinitions, concepts, syntax structures, semantics, and decodingprocesses of H.264/AVC apply also to MVC as such or with certaingeneralizations or constraints. Some definitions, concepts, syntaxstructures, semantics, and decoding processes of MVC are described inthe following.

An access unit in MVC is defined to be a set of NAL units that areconsecutive in decoding order and contain exactly one primary codedpicture consisting of one or more view components. In addition to theprimary coded picture, an access unit may also contain one or moreredundant coded pictures, one auxiliary coded picture, or other NALunits not containing slices or slice data partitions of a coded picture.The decoding of an access unit results in one decoded picture consistingof one or more decoded view components, when decoding errors, bitstreamerrors or other errors which may affect the decoding do not occur. Inother words, an access unit in MVC contains the view components of theviews for one output time instance.

A view component in MVC is referred to as a coded representation of aview in a single access unit.

Inter-view prediction may be used in MVC and refers to prediction of aview component from decoded samples of different view components of thesame access unit. In MVC, inter-view prediction is realized similarly tointer prediction. For example, inter-view reference pictures are placedin the same reference picture list(s) as reference pictures for interprediction, and a reference index as well as a motion vector are codedor inferred similarly for inter-view and inter reference pictures.

An anchor picture is a coded picture in which all slices may referenceonly slices within the same access unit, i.e., inter-view prediction maybe used, but no inter prediction is used, and all following codedpictures in output order do not use inter prediction from any pictureprior to the coded picture in decoding order. Inter-view prediction maybe used for IDR view components that are part of a non-base view. A baseview in MVC is a view that has the minimum value of view order index ina coded video sequence. The base view can be decoded independently ofother views and does not use inter-view prediction. The base view can bedecoded by H.264/AVC decoders supporting only the single-view profiles,such as the Baseline Profile or the High Profile of H.264/AVC.

In the MVC standard, many of the sub-processes of the MVC decodingprocess use the respective sub-processes of the H.264/AVC standard byreplacing term “picture”, “frame”, and “field” in the sub-processspecification of the H.264/AVC standard by “view component”, “frame viewcomponent”, and “field view component”, respectively. Likewise, terms“picture”, “frame”, and “field” are often used in the following to mean“view component”, “frame view component”, and “field view component”,respectively.

In scalable multiview coding, the same bitstream may contain coded viewcomponents of multiple views and at least some coded view components maybe coded using quality and/or spatial scalability.

Many video encoders utilize the Lagrangian cost function to findrate-distortion optimal coding modes, for example the desired macroblockmode and associated motion vectors. This type of cost function uses aweighting factor or (lambda) to tie together the exact or estimatedimage distortion due to lossy coding methods and the exact or estimatedamount of information required to represent the pixel/sample values inan image area. The Lagrangian cost function may be represented by theequation:

C=DλR

where C is the Lagrangian cost to be minimized, D is the imagedistortion (for example, the mean-squared error between the pixel/samplevalues in original image block and in coded image block) with the modeand motion vectors currently considered, is a Lagrangian coefficient andR is the number of bits needed to represent the required data toreconstruct the image block in the decoder (including the amount of datato represent the candidate motion vectors).

In the following, the term layer is used in context of any type ofscalability, including view scalability and depth enhancements. Anenhancement layer refers to any type of an enhancement, such as SNR,spatial, multiview, depth, bit-depth, chroma format, and/or color gamutenhancement. A base layer also refers to any type of a base operationpoint, such as a base view, a base layer for SNR/spatial scalability, ora texture base view for depth-enhanced video coding.

Scalable video (de)coding may be realized with a concept known assingle-loop decoding, where decoded reference pictures are reconstructedonly for the highest layer being decoded while pictures at lower layersmay not be fully decoded or may be discarded after using them forinter-layer prediction. In single-loop decoding, the decoder performsmotion compensation and full picture reconstruction only for thescalable layer desired for playback (called the “desired layer” or the“target layer”), thereby reducing decoding complexity when compared tomulti-loop decoding. All of the layers other than the desired layer donot need to be fully decoded because all or part of the coded picturedata is not needed for reconstruction of the desired layer. However,lower layers (than the target layer) may be used for inter-layer syntaxor parameter prediction, such as inter-layer motion prediction.Additionally or alternatively, lower layers may be used for inter-layerintra prediction and hence intra-coded blocks of lower layers may haveto be decoded. Additionally or alternatively, inter-layer residualprediction may be applied, where the residual information of the lowerlayers may be used for decoding of the target layer and the residualinformation may need to be decoded or reconstructed. In some codingarrangements, a single decoding loop is needed for decoding of mostpictures, while a second decoding loop may be selectively applied toreconstruct so-called base representations (i.e. decoded base layerpictures), which may be needed as prediction references but not foroutput or display.

There are ongoing standardization activities to specify a multiviewextension of HEVC (which may be referred to as MV-HEVC), adepth-enhanced multiview extension of HEVC (which may be referred to as3D-HEVC), and a scalable extension of HEVC (which may be referred to asSHVC). A multi-loop decoding operation has been envisioned to be used inall these specifications.

In scalable video coding schemes utilizing multi-loop (de)coding,decoded reference pictures for each (de)coded layer may be maintained ina decoded picture buffer (DPB). The memory consumption for DPB maytherefore be significantly higher than that for scalable video codingschemes with single-loop (de)coding operation. However, multi-loop(de)coding may have other advantages, such as relatively few additionalparts compared to single-layer coding.

In scalable video coding with multi-loop decoding, enhanced layers maybe predicted from pictures that had been already decoded in the base(reference) layer. Such pictures may be stored in the DPB of base layerand may be marked as used for reference. In certain circumstances, apicture marked as used for reference may be stored in fast memory, inorder to provide fast random access to its samples, and may remainstored after the picture is supposed to be displayed in order to be usedas reference for prediction. This imposes requirements on memoryorganization. In order to relax such memory requirements, a conventionaldesign in multi-loop multilayer video coding schemes (such as MVC)assumes restricted utilization of inter-layer predictions.Inter-layer/inter-view prediction for enhanced view is allowed from adecoded picture of the base view located at the same access unit, inother word representing the scene at the same time entity. In suchdesigns, the number of reference pictures available for predictingenhanced views is increased by 1 for each reference view.

It has been proposed that in scalable video coding with multi-loop(de)coding operation pictures marked as used for reference need notoriginate from the same access units in all layers. For example, asmaller number of reference pictures may be maintained in an enhancementlayer compared to the base layer. In some embodiments a temporalinter-layer prediction, which may also be referred to as a diagonalinter-layer prediction or diagonal prediction, can be used to improvecompression efficiency in such coding scenarios. In general, diagonalprediction may refer to any prediction where the prediction crosses morethan one scalability domain or scalability type. For example, diagonalprediction may refer to prediction that takes place from a differentcomponent type (e.g. from depth to texture) and from a different timeinstant (e.g. from a picture of a previous access unit in (de)codingorder to a picture in the current access unit).

A decoding process may be specified with reference to a layer identifierlist TargetDecLayerIdList, which specifies the list of layer identifiervalues, such as nuh_layer_id values. The layer identifier values may bein TargetDecLayerIdList in increasing order of the NAL units to bedecoded. TargetDecLayerIdList may include the layer identifiers forlayers that are intended to be output by the decoder as well as all thelayers on which the output layers depend in the decoding process.

Work is ongoing to specify scalable and multiview extensions to the HEVCstandard. The multiview extension of HEVC, referred to as MV-HEVC, issimilar to the MVC extension of H.264/AVC. Similarly to MVC, in MV-HEVC,inter-view reference pictures can be included in the reference picturelist(s) of the current picture being coded or decoded. The scalableextension of HEVC, referred to as SHVC, is planned to be specified sothat it uses multi-loop decoding operation (unlike the SVC extension ofH.264/AVC). Currently, two designs to realize scalability areinvestigated for SHVC. One is reference index based, where aninter-layer reference picture can be included in a one or more referencepicture lists of the current picture being coded or decoded (asdescribed above). Another may be referred to as IntraBL or TextureRL,where a specific coding mode, e.g. in CU level, is used for usingdecoded/reconstructed sample values of a reference layer picture forprediction in an enhancement layer picture. The SHVC development hasconcentrated on development of spatial and coarse grain qualityscalability.

It is possible to use many of the same syntax structures, semantics, anddecoding processes for MV-HEVC and reference-index-based SHVC.Furthermore, it is possible to use the same syntax structures,semantics, and decoding processes for depth coding too. Hereafter, termscalable multiview extension of HEVC (SMV-HEVC) is used to refer to acoding process, a decoding process, syntax, and semantics where largelythe same (de)coding tools are used regardless of the scalability typeand where the reference index based approach without changes in thesyntax, semantics, or decoding process below the slice header is used.SMV-HEVC might not be limited to multiview, spatial, and coarse grainquality scalability but may also support other types of scalability,such as depth-enhanced video.

For the enhancement layer coding, the same concepts and coding tools ofHEVC may be used in SHVC, MV-HEVC, and/or SMV-HEVC. However, theadditional inter-layer prediction tools, which employ already coded data(including reconstructed picture samples and motion parameters a.k.amotion information) in reference layer for efficiently coding anenhancement layer, may be integrated to SHVC, MV-HEVC, and/or SMV-HEVCcodec.

In MV-HEVC, SMV-HEVC, and reference index based SHVC solution, the blocklevel syntax and decoding process are not changed for supportinginter-layer texture prediction. Only the high-level syntax has beenmodified (compared to that of HEVC) so that reconstructed pictures(upsampled if necessary) from a reference layer of the same access unitcan be used as the reference pictures for coding the current enhancementlayer picture. The inter-layer reference pictures as well as thetemporal reference pictures are included in the reference picture lists.The signalled reference picture index is used to indicate whether thecurrent Prediction Unit (PU) is predicted from a temporal referencepicture or an inter-layer reference picture. The use of this feature maybe controlled by the encoder and indicated in the bitstream for examplein a video parameter set, a sequence parameter set, a picture parameter,and/or a slice header. The indication(s) may be specific to anenhancement layer, a reference layer, a pair of an enhancement layer anda reference layer, specific TemporalId values, specific picture types(e.g. IRAP pictures), specific slice types (e.g. P and B slices but notI slices), pictures of a specific POC value, and/or specific accessunits, for example. The scope and/or persistence of the indication(s)may be indicated along with the indication(s) themselves and/or may beinferred.

The reference list(s) in MV-HEVC, SMV-HEVC, and a reference index basedSHVC solution may be initialized using a specific process in which theinter-layer reference picture(s), if any, may be included in the initialreference picture list(s). are constructed as follows. For example, thetemporal references may be firstly added into the reference lists (L0,L1) in the same manner as the reference list construction in HEVC. Afterthat, the inter-layer references may be added after the temporalreferences.

The inter-layer reference pictures may be for example concluded from thelayer dependency information, such as the RefLayerId[i] variable derivedfrom the VPS extension as described above. The inter-layer referencepictures may be added to the initial reference picture list L0 if thecurrent enhancement-layer slice is a P-Slice, and may be added to bothinitial reference picture lists L0 and L1 if the currentenhancement-layer slice is a B-Slice. The inter-layer reference picturesmay be added to the reference picture lists in a specific order, whichcan but need not be the same for both reference picture lists. Forexample, an opposite order of adding inter-layer reference pictures intothe initial reference picture list 1 may be used compared to that of theinitial reference picture list 0. For example, inter-layer referencepictures may be inserted into the initial reference picture 0 in anascending order of nuh_layer_id, while an opposite order may be used toinitialize the initial reference picture list 1.

In the coding and/or decoding process, the inter-layer referencepictures may be treated as a long-term reference pictures.

In SMV-HEVC and a reference index based SHVC solution, inter-layermotion parameter prediction may be performed by setting the inter-layerreference picture as the collocated picture for TMVP derivation. Amotion field mapping process between two layers may be performed forexample to avoid block level decoding process modification in TMVPderivation. A motion field mapping could also be performed for multiviewcoding, but a present draft of MV-HEVC does not include such a process.The use of the motion field mapping feature may be controlled by theencoder and indicated in the bitstream for example in a video parameterset, a sequence parameter set, a picture parameter, and/or a sliceheader. The indication(s) may be specific to an enhancement layer, areference layer, a pair of an enhancement layer and a reference layer,specific TemporalId values, specific picture types (e.g. RAP pictures),specific slice types (e.g. P and B slices but not I slices), pictures ofa specific POC value, and/or specific access units, for example. Thescope and/or persistence of the indication(s) may be indicated alongwith the indication(s) themselves and/or may be inferred.

In a motion field mapping process for spatial scalability, the motionfield of the upsampled inter-layer reference picture is attained basedon the motion field of the respective reference layer picture. Themotion parameters (which may e.g. include a horizontal and/or verticalmotion vector value and a reference index) and/or a prediction mode foreach block of the upsampled inter-layer reference picture may be derivedfrom the corresponding motion parameters and/or prediction mode of thecollocated block in the reference layer picture. The block size used forthe derivation of the motion parameters and/or prediction mode in theupsampled inter-layer reference picture may be for example 16×16. The16×16 block size is the same as in HEVC TMVP derivation process wherecompressed motion field of reference picture is used.

In a textureRL based SHVC solution, the inter-layer texture predictionmay be performed at CU level for which a new prediction mode, named astextureRL mode, is introduced. The collocated upsampled base layer blockis used as the prediction for the enhancement layer CU coded intextureRL mode. For an input CU of the enhancement layer encoder, the CUmode may be determined among intra, inter and textureRL modes, forexample. The use of the textureRL feature may be controlled by theencoder and indicated in the bitstream for example in a video parameterset, a sequence parameter set, a picture parameter, and/or a sliceheader. The indication(s) may be specific to an enhancement layer, areference layer, a pair of an enhancement layer and a reference layer,specific TemporalId values, specific picture types (e.g. RAP pictures),specific slice types (e.g. P and B slices but not I slices), pictures ofa specific POC value, and/or specific access units, for example. Thescope and/or persistence of the indication(s) may be indicated alongwith the indication(s) themselves and/or may be inferred. Furthermore,the textureRL may be selected by the encoder at CU level and may beindicated in the bitstream per each CU for example using a CU level flag(texture_rl_flag) which may be entropy-coded e.g. using context adaptivearithmetic coding (e.g. CABAC).

The residue of textureRL predicted CU may be coded as follows. Thetransform process of textureRL predicted CU may be the same as that forthe intra predicted CU, where a discrete sine transform (DST) is appliedto TU of luma component having 4×4 size and a discrete cosine transform(DCT) is applied to the other type of TUs. Transform coefficient codingof a textureRL-predicted CU may be the same to that of inter predictedCU, where no_residue_flag may be used to indicate whether thecoefficients of the whole CU are skipped.

In a textureRL based SHVC solution, in addition to spatially andtemporally neighboring PUs, the motion parameters of the collocatedreference-layer block may also be used to form the merge candidate list.The base layer merge candidate may be derived at a location collocatedto the central position of the current PU and may be inserted in aparticular location of the merge list, such as the first candidate inmerge list. In the case of spatial scalability, the reference-layermotion vector may be scaled according to the spatial resolution ratiobetween the two layers. The pruning (duplicated candidates check) may beperformed for each spatially neighboring candidate with collocated baselayer candidate. For the collocated base layer merge candidate andspatial merge candidate derivation, a certain maximum number of mergecandidates may be used; for example four merge candidates may beselected among candidates that are located in six different positions.The temporal merge candidate may be derived in the same manner as donefor HEVC merge list. When the number of candidates does not reach tomaximum number of merge candidates (which may be determined by theencoder and may be indicated in the bitstream and may be assigned to thevariable MaxNumMergeCand), the additional candidates, including combinedbi-predictive candidates and zero merge candidates, may be generated andadded at the end of the merge list, similarly or identically to HEVCmerge list construction.

In some coding and/or decoding arrangements, a reference index basedscalability and a block-level scalability approach, such a textureRLbased approach, may be combined. For example, multiview-video-plus-depthcoding and/or decoding may be performed as follows. A textureRL approachmay be used between the components of the same view. For example, adepth view component may be inter-layer predicted using a textureRLapproach from a texture view component of the same view. A referenceindex based approach may be used for inter-view prediction, and in someembodiments inter-view prediction may be applied only between viewcomponents of the same component type.

Work is also ongoing to specify depth-enhanced video coding extensionsto the HEVC standard, which may be referred to as 3D-HEVC, in whichtexture views and depth views may be coded into a single bitstream wheresome of the texture views may be compatible with HEVC. In other words,an HEVC decoder may be able to decode some of the texture views of sucha bitstream and can omit the remaining texture views and depth views.

Other types of scalability and scalable video coding include bit-depthscalability, where base layer pictures are coded at lower bit-depth(e.g. 8 bits) per luma and/or chroma sample than enhancement layerpictures (e.g. 10 or 12 bits), chroma format scalability, whereenhancement layer pictures provide higher fidelity and/or higher spatialresolution in chroma (e.g. coded in 4:4:4 chroma format) than base layerpictures (e.g. 4:2:0 format), and color gamut scalability, where theenhancement layer pictures have a richer/broader color representationrange than that of the base layer pictures—for example the enhancementlayer may have UHDTV (ITU-R BT.2020) color gamut and the base layer mayhave the ITU-R BT.709 color gamut. Any number of such other types ofscalability may be realized for example with a reference index basedapproach or a block-based approach e.g. as described above.

A way of categorizing different types of prediction is to consideracross which domains or scalability types the prediction crosses. Thiscategorization may lead into one or more of the following types ofprediction, which may also sometimes be referred to as predictiondirections:

-   -   Temporal prediction e.g. of sample values or motion vectors from        an earlier picture usually of the same scalability layer, view        and component type (texture or depth).    -   Inter-view prediction (which may be also referred to as        cross-view prediction) referring to prediction taking place        between view components usually of the same time instant or        access unit and the same component type.    -   Inter-layer prediction referring to prediction taking place        between layers usually of the same time instant, of the same        component type, and of the same view.    -   Inter-component prediction may be defined to comprise prediction        of syntax element values, sample values, variable values used in        the decoding process, or anything alike from a component picture        of one type to a component picture of another type. For example,        inter-component prediction may comprise prediction of a texture        view component from a depth view component, or vice versa.

Prediction approaches using image information from a previously codedimage can also be called as inter prediction methods. Inter predictionmay sometimes be considered to only include motion-compensated temporalprediction, while it may sometimes be considered to include all types ofprediction where a reconstructed/decoded block of samples is used asprediction source, therefore including conventional inter-viewprediction for example. Inter prediction may be considered to compriseonly sample prediction but it may alternatively be considered tocomprise both sample and syntax prediction. As a result of syntax andsample prediction, a predicted block of pixels of samples may beobtained.

If the prediction, such as predicted variable values and/or predictionblocks, is not refined by the encoder using any form of prediction erroror residual coding, prediction may be referred to as inheritance. Forexample, in the merge mode of HEVC, the prediction motion information isnot refined e.g. by (de)coding motion vector differences, and hence themerge mode may be considered as an example of motion informationinheritance.

The several embodiments of the invention relates to scalable videocoding, including spatial, quality, multiview and/or depth scalability.Video coding schemes may utilize a prediction scheme between pictures.As discussed, prediction may be performed in the encoder for examplethrough a process of block partitioning and block matching between acurrently coded block (Cb) in the current picture and a reference block(Rb) in the picture which is selected as a reference. Thereforeparameters of such prediction can be defined as motion information (MI)comprising for example on or more of the following: spatial coordinatesof the Cb (e.g. coordinates of the top-left pixel of the Cb); areference index refIdx or similar which specifies the picture in thereference picture list which is selected as reference picture; a motionvector (MV) specifying displacement between the spatial coordinated ofthe Cb and Rb in the reference picture; and the size and shape of themotion partition (the size and shape of the matching block).

A motion field associated with a picture may be considered to comprise aset of motion information produced for every coded block of the picture.A motion field may be accessible by coordinates of a block, for example.A motion field may be used for example in Temporal motion vectorprediction or any other motion prediction mechanism where a source or areference for prediction other than the current decoded/coded picture isused.

Video coding schemes may utilize a temporal motion vector predictionscheme, such as the temporal direct mode in H.264/AVC or the temporalmotion vector predictor (TMVP) candidate in the merge and AVMP modes ofH.265/HEVC. In a temporal motion vector prediction scheme, at least asubset of the motion information of another picture is used to derivemotion information or motion information predictor(s) for the currentpicture. Temporal motion vector prediction therefore requires storage ofmotion information of reference pictures.

In H.265/HEVC, the sequence parameter set includes thesps_temporal_mvp_enabled_flag syntax element, which indicates if theslice header includes the slice_temporal_mvp_enabled_flag. Ifsps_temporal_mvp_enabled_flag is equal to 0, no temporal motion vectorpredictors are used in the coded video sequence.slice_temporal_mvp_enabled_flag specifies whether temporal motion vectorpredictors can be used for inter prediction. Whenslice_temporal_mvp_enabled_flag is equal to 1, there are syntax elementsin the slice header that identify the collocated picture used to derivethe temporal motion vector predictors.

Temporal motion vector prediction can also be used in scalable videocoding when a motion field of an inter-layer reference picture is usedto predict or derive motion information of the current picture.

Motion field mapping may be used for example when an inter-layerreference picture is of different spatial resolution than the currentpicture. In a motion field mapping process for spatial scalability, themotion field of the upsampled inter-layer reference picture is attainedbased on the motion field of the respective reference layer picture. Themotion parameters (which may e.g. include a horizontal and/or verticalmotion vector value and a reference index) and/or a prediction mode foreach block of the upsampled inter-layer reference picture may be derivedfrom the corresponding motion parameters and/or prediction mode of thecollocated block in the reference layer picture.

The storage of motion information may be performed for example on thebasis of the minimum size of a motion partition, e.g. 4×4 (of lumasamples) in the case of H.264/AVC. In another example, the spatialgranularity of motion information may be pre-defined for example in acoding standard and the coded motion information may be resampled orconverted to that spatial granularity. For example, motion informationcan be stored for 16×16 blocks (of luma samples) in H.265/HEVC.

It is known to add a flag in SPS extension to indicate constraint oncollocated picture, such that the collocated picture is an inter-layerreference picture rather than a temporal reference pictures for non-baselayer coding. As a result, the motion information of temporal referencepictures within current enhancement layer need not be stored. Thesemantics of the flag was proposed to be the following:collocated_picture_constraint_flag equal to 1 specifies that collocatedpicture used for inter-layer prediction are constrained in the CVS. Whencollocated_picture_constraint_flag is equal to 1, the collocated pictureused for inter-layer prediction shall be the reference picture withnuh_layer_id not equal to that of current picture. Whencollocated_picture_constraint_flag is equal to 0, no constraint for thecollocated picture used for inter-layer prediction is signaled by thisflag. When not present, the collocated_picture_constraint_flag isinferred to be equal to 0.

Codec implementations may have different approaches for storing ofmotion information. Assuming that motion information is stored for aregular grid of blocks, one example approach may be to store for eachblock in the grid the following:

-   -   1. Indication of the number of motion-compensated prediction        blocks (2 bits). 0 indicates that there are no prediction blocks        (e.g. when intra coding was used), 1 indicates uni-prediction,        and 2 indicates bi-prediction;    -   2. Reference picture list indices for reference picture list 0        and list 1. In H.265/HEVC, the index may be limited for example        to 0 to 15, inclusive, and hence 8 bits is needed for reference        index storage. Additionally, reference picture lists have to be        stored for each slice of the picture.    -   3. Horizontal and vertical motion vector components for the        motion vector (uni-prediction) or the tow motion vectors        (bi-prediction). The number of bits required to store a motion        vector component may depend for example on level limits, picture        extents, and/or specific constraints that may be for example        inferred by a coding profile of specifically indicated in the        bitstream. Generally, for example 2 bytes per motion vector        component may be needed

Assuming that no. 1 above can be stored in spare bits left over by amotion vector components not requiring a full two bytes for storage, itmay take for example 9 bytes to store motion information for a singleblock. In 16×16 motion block size (of luma samples), a motion fieldusing 9 bytes per block for motion information is about 2.3% of thestorage space needed to store sample values of a reference picture of4:2:0 chroma format.

For example in H.264/AVC, it may be sufficient to store only referenceindex to reference picture list 0 (if present) or to reference picturelist 1 otherwise and only one pair of horizontal and vertical motionvector components. It may therefore take 5 bytes to store motioninformation per block in a motion field. In 4×4 motion block size (ofluma samples), a motion field using 5 bytes per block for motioninformation is about 20.8% of the storage space needed to store samplevalues of a reference picture of 4:2:0 chroma format.

As can be understood form these examples on the storage space usage, theamount of memory used for motion field storage can be substantial, andhence it would be beneficial to indicate which motion fields areactually needed for a temporal motion vector prediction mechanism oralike, such as the TMVP mechanism of H.265/HEVC or alike, in order toallow optimized memory usage in decoders.

In H.265/HEVC, the sps_temporal_mvp_enabled_flag indicates whether theTMVP mechanism may be in use (when the flag is equal to 1) or is not inuse (when the flag is equal to 0) in the HEVC base layer/view (withnuh_layer_id equal to 0). When sps_temporal_mvp_enabled_flag is equal to1, the slice_temporal_mvp_enabled_flag is present in the slice headerand indicates if the TMVP mechanism is in use for the current picture.

There may be “black box” implementations of scalable extensions of HEVC,where the base layer decoding/coding is implemented with an existingHEVC v1 implementation without changes. Such an implementation of baselayer decoding/coding would store motion fields only ifsps_temporal_mvp_enabled_flag is equal to 1.

Base layer motion fields may be used for either or both of the followingtwo purposes: temporal motion vector prediction between pictures of thebase layer and inter-layer motion prediction. If the base layer motionfields are used only for inter-layer motion prediction, the memory usedfor base layer motion fields could be de-allocated or used for otherpurposes after decoding of the access unit has been finished (or, moreaccurately, decoding of all layers within the access unit that may usethe abase layer as motion prediction reference has been finished).However, when sps_temporal_mvp_enabled_flag is used to control thestorage of base layer motion fields, it cannot be used to indicate thatbase layer motion fields are used only for inter-layer motion predictionand not for temporal motion vector prediction within the base layer.

If a first enhancement layer for which thecollocated_picture_constraint_flag (as described above) is set equal to1 is used as a reference for inter-layer prediction for a secondenhancement layer, the storage of the motion field of pictures in thefirst enhancement layer may be needed for inter-layer motion prediction.In other words, the described collocated_picture_constraint_flag is notsufficient to determine if the motion field of the enhancement layerpicture needs to be stored or not.

While the known approaches aim at controlling the motion field memoryusage for entire layers within a sequence, a finer granularity within alayer may be desired and/or indications to reduce the needed storagespace of a motion field may be desired. The total memory use for motionfield storage may then be controlled and may be smaller than the memoryuse assuming a maximum amount of reference pictures for each of which amotion field is stored.

The present embodiments allow the encoder to indicate or control themotion field storage for the decoder in one or more of the followingways:

-   -   a) Motion fields of the base layer are to be stored but may only        be needed for inter-layer motion prediction (and are not needed        for motion prediction for any other picture in the base layer).        Discussed in more detailed in section “A”.;    -   b) Indications related to motion field storage and/or use reside        in sequence-level syntax structures but may be limited to apply        to only some picture within a layer. Discussed in more detailed        in section “B”.;    -   c) The storage or marking of a motion field of a picture may be        controlled with a later picture for example using reference        motion field set syntax. The storage or marking of a motion        field may differ from the storage or corresponding marking of a        corresponding reference picture. Discussed in more detailed in        section “C”.;    -   d) The storage space required by a motion field may be        controlled by the encoder with various indicated constraints.        Discussed in more detailed in section “D”.

Section A) Sequence-Level Indications for Motion Field Storage

In some embodiments, two or more of the following are indicated by theencoder in the bitstream and/or decoded by the decoder from thebitstream: 1) An indication capable of indicating if motion fields maybe used within a layer for temporal motion vector prediction or are notused within a layer for temporal motion vector prediction. Thisindication may be layer-wise; 2) An indication capable of indicating ifinter-layer motion prediction may be used or if inter-layer motionprediction is not used. This indication may be specific to a pair oflayers; 3) An indication capable of indicating if diagonal motionprediction may be used or if diagonal motion prediction is not used.This indication may be specific to pair of layers.

For example, in the context of H.265/HEVC and/or its extensions, anindication capable of indicating if motion fields may be used within alayer for temporal motion vector prediction or are not used within alayer for temporal motion vector prediction may be specified for examplein the SPS extension. For example, the following syntax may be used:

Descriptor sps_extension( ) {  ...  collocated_picture_constraint_flagu(1)  ... }

The semantics of collocated_picture_constraint_flag may be specified forexample as follows or in any similar manner.collocated_picture_constraint_flag equal to 1 indicates the following:

-   -   For coded slice segment NAL units with nuh_layer_id equal to 0,        collocated_picture_constraint_flag equal to 1 indicates that        slice_temporal_mvp_enabled_flag is not present or is equal to 0.    -   For coded slice segment NAL units with nuh_layer_id greater than        0, collocated_picture_constraint_flag equal to 1 indicates that        the collocated picture, if any, is an inter-layer reference        picture.

It should be understood that the semantics ofcollocated_picture_constraint_flag or alike may be specifiedequivalently with other phrasing. For example, the following phrasingcould be used: collocated_picture_constraint_flag equal to 1 indicatesthe following:

-   -   In coded slice segment NAL units with nuh_layer_id equal to 0        for which this SPS is the active SPS,        slice_temporal_mvp_enabled_flag is not present or is equal to 0.    -   In coded slice segment NAL units with nuh_layer_id nuhLayerIdA        greater 0 for which this SPS is the active layer SPS, the        collocated picture, if any, has nuh_layer_id nuhLayerIdB that is        not equal to nuhLayerIdA

When collocated_picture_constraint_flag is equal to 0, no constraint forthe collocated picture is indicated. When not present, thecollocated_picture_constraint_flag is inferred to be equal to 0.

The collocated_picture_constraint_flag or alike may be used, for examplefor determining if storing of motion fields is needed, together with anindication capable of indicating if inter-layer motion prediction may beused or is not used. For example, in the context of H.265/HEVC and/orits extensions, an indication capable of indicating if inter-layermotion prediction may be used or if inter-layer motion prediction is notused may be performed through the direct_dependency_type[i][j] syntaxelement of the VPS extension.

In some embodiments, a decoder may store motion fields as part ofdecoding a picture when any of the indications 1 to 3 above indicatesthat motion prediction may be used.

In some embodiments, an encoder and/or a decoder may store motion fieldsas part of decoding/coding a current picture at a particular layer (e.g.layer A) when at least one of the following is true:

-   -   It is indicated in the bitstream that temporal motion vector        prediction may be used within the layer.    -   It is indicated in the bitstream that inter-layer and/or        diagonal prediction may be used and the pictures that may use        the current picture as a reference for motion prediction may be        or are present in the bitstream and are intended to be decoded.

For example, if the motion field of the picture is indicated not to beused for motion prediction for any other picture at layer A, and iflayer A may be used as reference for inter-layer and/or diagonal motionprediction for other layers, but none of those other layers are presentin the bitstream or are not intended to be decoded, the motion field ofthe picture (at layer A) need not be stored and decoders may determinenot to store the motion field of the picture (at layer A). Otherwise,the motion field of the picture (at layer A) may be needed forinter-layer and/or diagonal motion prediction for other layers and hencedecoders store the motion field.

The encoder and/or the decoder may conclude that the motion field or themotion vectors of the current picture have to be stored when they may beused for temporal motion vector prediction of other pictures in the samelayer or when they may be used for inter-layer motion prediction. WhenHEVC and/or HEVC extensions are used, the encoder and/or the decoder mayconclude that the motion field or the motion vectors of the currentpicture have to be stored when at least one of the following is true:

-   -   sps_temporal_mvp_enabled_flag is equal to 1 and        collocated_picture_constraint_flag is equal 0.    -   collocated_picture_constraint_flag is equal to 1 and there is a        nuh_layer_id value nuhLayerIdA such that        MotionPredRefLayerId[nuhLayerIdA][mIdx] for any mIdx in the        range of 0 to NumMotionPredRefLayers[nuhLayerIdA]−1, inclusive,        is equal to the nuh_layer_id value of the current picture and        nuhLayerIdA is among TargetDecLayerIdList.

The encoder and/or the decoder may conclude that the motion field or themotion vectors of a picture need no longer be stored when either thepicture is marked as “unused for reference” or when the picture is notused for temporal motion vector prediction of other pictures in the samelayer and all pictures in the same access unit that may use the pictureas a reference for inter-layer motion prediction have been decoded. WhenHEVC and/or HEVC extensions are used, the encoder and/or the decoder mayconclude that the motion field or the motion vectors of a picture needno longer be stored when at least one of the following is true:

-   -   The picture is marked as “unused for reference”.    -   collocated_picture_constraint_flag is equal to 1 for the picture        and the access unit containing the picture has been decoded.    -   collocated_picture_constraint_flag is equal to 1 for the picture        and there is no nuh_layer_id value nuhLayerIdA among those        values of TargetDecLayerIdList that remain to be decoded from        the access unit containing the picture such that        MotionPredRefLayerId[nuhLayerIdA][mIdx] for any mIdx in the        range of 0 to NumMotionPredRefLayers[nuhLayerIdA]−1, inclusive,        is equal to the nuh_layer_id value of the picture.

Section B) Sequence-Level Sub-Layer-Wise Indications for Motion FieldStorage

In some embodiments, the indications relate to motion field storageand/or use reside in sequence-level syntax structures but may be limitedto apply to only some pictures within a layer. The limited scope may bedetermined and indicated in the bitstream by the encoder and decodedfrom the bitstream by the decoder. For example, the indications mayapply to only certain temporal sub-layers and/or picture types (such asIRAPs). Different scope may be selected for different indications or thesame type of indication for different layers or pairs of layers.

Section C) Flexible Control of which Motion Fields are Stored

In some embodiments, an encoder and/or decoder may deallocate a motionfield storage buffer or use a motion field storage buffer for storinganother motion field, when the motion field in the storage buffer is nolonger needed for temporal, inter-layer or diagonal motion prediction.In some embodiments, an encoder and/or a decoder may mark a motion fieldor a corresponding reference picture e.g. as “motion field not neededfor prediction” or anything alike, when the motion field in the storagebuffer is no longer needed for temporal, inter-layer or diagonal motionprediction. Likewise, when a motion field is decoded and stored, it orthe corresponding reference picture may be marked by the encoder and/orthe decoder as “motion field needed for prediction” or anything alike.

The maximum number of motion fields are stored and/or marked as “motionfield needed for prediction” may be limited. This number may bepre-defined for example in a coding standard and/or may be determined bythe encoder and indicated in the bitstream. The maximum number may bedecoded from the bitstream by the decoder. The maximum number may bespecific to temporal sub-layers and may be separately indicated by theencoder for one or more temporal sub-layers within bitstream. Themaximum number may be lower than the maximum number of referencepictures.

The motion fields that are stored and/or marked as “motion field neededfor prediction” may be inferred by the encoder and/or the decoder usinga specific algorithm and/or may be determined by the encoder andindicated by the encoder in the bitstream and decoded by the decoderfrom the bitstream

Said specific algorithm may be pre-defined for example in a codingstandard. In some embodiments, many algorithms may be specified forexample in a coding standard and/or many parameter values controllingthe algorithms may be used. The encoder may determine which of said manyalgorithms is in use and which parameter values controlling thealgorithm are used and indicate those in the bitstream. Likewise, thedecoder may decode which of said many algorithms is in use and whichparameter values controlling the algorithm are used from the bitstream.For example, an algorithm may realize first-in-first-out (FIFO)buffering of motion fields. An action according to said algorithm may betriggered for example when the number of occupied motion field storagebuffers reaches or is about to exceed the maximum number. For example,an action may be that a motion field selected according to FIFObuffering in is deallocated or marked as “motion field not needed forprediction”.

In some embodiments, the encoder may encode in the bitstream and/or thedecoder may decode from the bitstream commands similar to MMCO ofH.264/AVC for controlling motion field marking.

In some embodiments, the encoder may encode in the bitstream and/or thedecoder may decode from the bitstream a reference motion field set(RMFS) similarly to a reference picture set (RPS) in H.265/HEVC. An RFMSsyntax structure may include syntax elements from which identifiers ofmotion fields included in RFMS may be concluded. The identifiers may forexample be picture identifiers, such as picture order counts or a partthereof (e.g. a certain number of the least significant bits of the POCvalues), picture order count differences (e.g. compared to the pictureorder count of the current picture), and/or frame_num values or similar.An RFMS syntax structure may for example have the same syntax as an RPSsyntax structure but an RFMS syntax structure may apply to a motionfields rather than decoded sample arrays (to which an RPS syntaxstructure may apply). The encoder and/or the decoder may keep thosemotion fields that are included in the RFMS in the memory (e.g. in adecoded motion field buffer) and/or marked as “motion field needed forprediction” (or alike). The encoder and/or the decoder may remove thosemotion fields that are not included in the RFMS from the memory (e.g.from a decoded motion field buffer) and/or mark them as “motion fieldnot needed for prediction” (or alike).

In some embodiments, the syntax of RPS is appended to indicate whetheror not a motion field is stored with the decoded picture included in theRPS. For example, a flag may be included for each picture in the RPS,which specifies that, when the flag is 1, the storage or presence of themotion field or that, when the flag is 0, the absence of the motionfield. The encoder and/or the decoder may store those motion fields thatare indicated to be present and/or mark them as “motion field needed forprediction” (or alike). The encoder and/or the decoder may remove thosemotion fields that are indicated to be absent and/or mark them as“motion field not needed for prediction” (or alike).

The encoder may be required to keep the number of motion fields markedas “motion field not needed for prediction” (or alike) according to RFMSor RPS not higher than the maximum number of motion fields.

In some embodiments, the encoder may encode in the bitstream and/or thedecoder may decode from the bitstream a first picture-wise indication,for example residing in a slice header, whether the picture may be usedas collocated picture for TMVP or alike for any subsequent picture inthe same layer. Alternatively or in addition, in some embodiments, theencoder may encode in the bitstream and/or the decoder may decode fromthe bitstream a second picture-wise indication, for example residing ina slice header, whether the picture may be used as a collocated picturefor TMVP or alike for any subsequent picture in the same layer or otherlayers. In some embodiments, the first and/or second picture-wiseindication may be used to control the storage and/or marking of themotion field or the corresponding picture. For example, if the secondindication is false (i.e. the picture is not used as collocated picturefor TMVP or alike for any subsequent picture), the motion field or thecorresponding picture may be marked as “motion field not needed forprediction”. In another example, if the first indication is false (i.e.the picture is not used as collocated for TMVP or alike for anysubsequent picture in the same layer), the motion field or thecorresponding picture may be marked as “motion field needed forprediction” until all pictures which may use the picture as collocatedpicture have been decoded after which the motion field or thecorresponding picture may be marked as “motion field not needed forprediction”. For example, a picture where the first indication is falsemay be marked as “motion field not needed for prediction” after theaccess unit in which the picture resides has been decoded (assuming nodiagonal prediction is in use).

In some embodiments, the first picture-wise indication and/or the secondpicture-wise indication as described above may be specific to certaintemporal sub-layers. For example, if the first or second picture-wiseindication is true for a picture with TemporalId equal to tIdA, thepicture may be used as collocated picture for TMVP or alike for thosesubsequent pictures within the same layer that have TemporalId greaterthan or equal to tIdA.

In some embodiments, it may be considered that motion fields are storedin a decoded motion field buffer (DMFB) or alike which may operatesimilarly to a decoded picture buffer. The encoder may encode in thebitstream and the decoder may decode from the bitstream syntax elementsand/or syntax element values that relate to the operation of the DMFB.Constraints may be specified for example in a coding standard for theDMFB, for example, when it comes to the maximum number of motion fieldbuffers in the DMFB and/or the maximum memory use of the DMFB.

Section D) Flexible Control of which Motion Information are Stored

In some embodiments, the encoder may indicate in the bitstream and thedecoder may decode from the bitstream a spatial relation or accuracy forstoring motion information. For example, a block size of luma samplesfor which the decoded/coded motion information is resampled may beindicated. For example, it may be indicated that motion information maybe stored for 32×32 blocks of luma samples. In some embodiments, theblock size may be spatially varying. In some embodiments, the block sizemay be selected according to prediction units of the correspondingpictures.

In some embodiments, the encoder may indicate in the bitstream and thedecoder may decode from the bitstream which parameters of the motioninformation are or may be needed in the motion prediction. For example,one or more of the following indications may be encoded by the encoderand/or decoded by the decoder:

-   -   Only inter-layer reference pictures are used for a particular        enhancement layer and hence motion vector values may be        restricted e.g. to (0, 0) in case of spatial and/or SNR        scalability and to (x, 0) for multiview coding with parallel        camera setup. Consequently, the motion field storage for that        enhancement layer can be tuned to store only a subset of motion        vector components.    -   Only inter-layer reference pictures are used (for sample        prediction) and only a certain reference layer may be used as        source for sample prediction for a particular enhancement layer        (e.g. as indicated by direct_dependency_type[i][j]).        Consequently, the motion field storage for that particular        enhancement layer can be tuned to omit reference index storage        or any other storage indicating which reference picture is used        for sample prediction.    -   Only uni-prediction is used.

In some embodiments, the encoder may infer or indicate in the bitstreamand the decoder may infer or decode from the bitstream constraints onparameters of the motion information which can help in reducing thestorage space for motion fields. For example, one or more of thefollowing may be inferred or encoded by the encoder and/or inferred ordecoded by the decoder:

-   -   Motion vector accuracy. For example, certain types of pictures        or layers, such as depth pictures or layers, may be inferred to        have motion vectors of integer luma sample accuracy and hence        motion fields for depth pictures need to be stored at        integer-sample accuracy. In another example, the encoder may        encode in the bitstream and the decoder may decode from the        bitstream and indication of motion vector accuracy. In another        example, a first layer, for example, representing a texture        view, may be used as motion prediction reference for another        layer, for example representing a depth view, but no motion        prediction within the first layer takes place. In this case, the        motion fields of the first layer may be quantized to a full        picture sample accuracy on the basis of the layers being of        different type or an indication by the encoder in the bitstream.    -   Maximum (absolute) horizontal and/or vertical motion vector        length.    -   Maximum reference index.    -   Maximum POC difference between a picture to which the motion        field corresponds and any used reference picture in the motion        field.

In some embodiments, it may be specified for example in a codingstandard how one or more of the above-mentioned or any other motionfield related parameter or parameter value or parameter constraintsaffects the memory use for a motion field. Such memory use may beconsidered in constraints specified for example for levels in a codingstandard and/or may affect the maximum number or motion field storagebuffers. Such memory use may be nominal and an encoder and/or a decoderimplementation may actually use another allocation of memory fordifferent parameters. For example, the memory use for a single motionvector component may be specified to be in bits log 2(2*maximum absolutemotion vector length/motion vector accuracy), where motion vectoraccuracy is e.g. ¼ for quarter-pixel motion vector accuracy, ½ forhalf-pixel motion vector accuracy, and so on. For example, if maximumabsolute motion vector length is 511 and motion vector accuracy is ¼, 12bits are needed to store the motion vector component. The memory use formotion information for a single block may be rounded up for example tofull bytes in order to provide easier memory access.

FIG. 1 shows a block diagram of a video coding system according to anexample embodiment as a schematic block diagram of an exemplaryapparatus or electronic device 50, which may incorporate a codecaccording to an embodiment of the invention. FIG. 2 shows a layout of anapparatus according to an example embodiment. The elements of FIGS. 1and 2 will be explained next.

The electronic device 50 may for example be a mobile terminal or userequipment of a wireless communication system. However, it would beappreciated that embodiments of the invention may be implemented withinany electronic device or apparatus which may require encoding anddecoding or encoding or decoding video images.

The apparatus 50 may comprise a housing 30 for incorporating andprotecting the device. The apparatus 50 further may comprise a display32 in the form of a liquid crystal display. In other embodiments of theinvention the display may be any suitable display technology suitable todisplay an image or video. The apparatus 50 may further comprise akeypad 34. In other embodiments of the invention any suitable data oruser interface mechanism may be employed. For example the user interfacemay be implemented as a virtual keyboard or data entry system as part ofa touch-sensitive display. The apparatus may comprise a microphone 36 orany suitable audio input which may be a digital or analogue signalinput. The apparatus 50 may further comprise an audio output devicewhich in embodiments of the invention may be any one of: an earpiece 38,speaker, or an analogue audio or digital audio output connection. Theapparatus 50 may also comprise a battery 40 (or in other embodiments ofthe invention the device may be powered by any suitable mobile energydevice such as solar cell, fuel cell or clockwork generator). Theapparatus may further comprise an infrared port 42 for short range lineof sight communication to other devices. In other embodiments theapparatus 50 may further comprise any suitable short range communicationsolution such as for example a Bluetooth wireless connection or aUSB/firewire wired connection.

The apparatus 50 may comprise a controller 56 or processor forcontrolling the apparatus 50. The controller 56 may be connected tomemory 58 which in embodiments of the invention may store both data inthe form of image and audio data and/or may also store instructions forimplementation on the controller 56. The controller 56 may further beconnected to codec circuitry 54 suitable for carrying out coding anddecoding of audio and/or video data or assisting in coding and decodingcarried out by the controller 56.

The apparatus 50 may further comprise a card reader 48 and a smart card46, for example a UICC and UICC reader for providing user informationand being suitable for providing authentication information forauthentication and authorization of the user at a network.

The apparatus 50 may comprise radio interface circuitry 52 connected tothe controller and suitable for generating wireless communicationsignals for example for communication with a cellular communicationsnetwork, a wireless communications system or a wireless local areanetwork. The apparatus 50 may further comprise an antenna 44 connectedto the radio interface circuitry 52 for transmitting radio frequencysignals generated at the radio interface circuitry 52 to otherapparatus(es) and for receiving radio frequency signals from otherapparatus(es).

In some embodiments of the invention, the apparatus 50 comprises acamera capable of recording or detecting individual frames which arethen passed to the codec 54 or controller for processing. In someembodiments of the invention, the apparatus may receive the video imagedata for processing from another device prior to transmission and/orstorage. In some embodiments of the invention, the apparatus 50 mayreceive either wirelessly or by a wired connection the image forcoding/decoding.

FIG. 3 shows an arrangement for video coding comprising a plurality ofapparatuses, networks and network elements according to an exampleembodiment. With respect to FIG. 3, an example of a system within whichembodiments of the present invention can be utilized is shown. Thesystem 10 comprises multiple communication devices which can communicatethrough one or more networks. The system 10 may comprise any combinationof wired or wireless networks including, but not limited to a wirelesscellular telephone network (such as a GSM, UMTS, CDMA network etc.), awireless local area network (WLAN) such as defined by any of the IEEE802.x standards, a Bluetooth personal area network, an Ethernet localarea network, a token ring local area network, a wide area network, andthe Internet.

The system 10 may include both wired and wireless communication devicesor apparatus 50 suitable for implementing embodiments of the invention.For example, the system shown in FIG. 3 shows a mobile telephone network11 and a representation of the internet 28. Connectivity to the internet28 may include, but is not limited to, long range wireless connections,short range wireless connections, and various wired connectionsincluding, but not limited to, telephone lines, cable lines, powerlines, and similar communication pathways.

The example communication devices shown in the system 10 may include,but are not limited to, an electronic device or apparatus 50, acombination of a personal digital assistant (PDA) and a mobile telephone14, a PDA 16, an integrated messaging device (IMD) 18, a desktopcomputer 20, a notebook computer 22. The apparatus 50 may be stationaryor mobile when carried by an individual who is moving. The apparatus 50may also be located in a mode of transport including, but not limitedto, a car, a truck, a taxi, a bus, a train, a boat, an airplane, abicycle, a motorcycle or any similar suitable mode of transport.

Some or further apparatuses may send and receive calls and messages andcommunicate with service providers through a wireless connection 25 to abase station 24. The base station 24 may be connected to a networkserver 26 that allows communication between the mobile telephone network11 and the internet 28. The system may include additional communicationdevices and communication devices of various types.

The communication devices may communicate using various transmissiontechnologies including, but not limited to, code division multipleaccess (CDMA), global systems for mobile communications (GSM), universalmobile telecommunications system (UMTS), time divisional multiple access(TDMA), frequency division multiple access (FDMA), transmission controlprotocol-internet protocol (TCP-IP), short messaging service (SMS),multimedia messaging service (MMS), email, instant messaging service(IMS), Bluetooth, IEEE 802.11 and any similar wireless communicationtechnology. A communications device involved in implementing variousembodiments of the present invention may communicate using various mediaincluding, but not limited to, radio, infrared, laser, cableconnections, and any suitable connection.

In the above, the example embodiments have been described with referenceto an encoder, it needs to be understood that the resulting bitstreamand the decoder have corresponding elements in them. Likewise, where theexample embodiments have been described with reference to a decoder, itneeds to be understood that the encoder has structure and/or computerprogram for generating the bitstream to be decoded by the decoder.

In the above, the example embodiments have been described with the helpof syntax of the bitstream. It needs to be understood, however, that thecorresponding structure and/or computer program may reside at theencoder for generating the bitstream and/or at the decoder for decodingthe bitstream.

In the above, some embodiments have been described in relation toparticular types of parameter sets. It needs to be understood, however,that embodiments could be realized with any type of parameter set orother syntax structure in the bitstream.

In the above, some embodiments have been described in relation toencoding indications, syntax elements, and/or syntax structures into abitstream or into a coded video sequence and/or decoding indications,syntax elements, and/or syntax structures from a bitstream or from acoded video sequence. It needs to be understood, however, thatembodiments could be realized when encoding indications, syntaxelements, and/or syntax structures into a syntax structure or a dataunit that is external from a bitstream or a coded video sequencecomprising video coding layer data, such as coded slices, and/ordecoding indications, syntax elements, and/or syntax structures from asyntax structure or a data unit that is external from a bitstream or acoded video sequence comprising video coding layer data, such as codedslices. For example, in some embodiments, an indication according to anyembodiment above may be coded into a video parameter set or a sequenceparameter set, which is conveyed externally from a coded video sequencefor example using a control protocol, such as SDP. Continuing the sameexample, a receiver may obtain the video parameter set or the sequenceparameter set, for example using the control protocol, and provide thevideo parameter set or the sequence parameter set for decoding.

Although the above examples describe embodiments of the inventionoperating within a codec within an electronic device, it would beappreciated that the invention as described below may be implemented aspart of any video codec. Thus, for example, embodiments of the inventionmay be implemented in a video codec which may implement video codingover fixed or wired communication paths.

Thus, user equipment may comprise a video codec such as those describedin embodiments of the invention above. It shall be appreciated that theterm user equipment is intended to cover any suitable type of wirelessuser equipment, such as mobile telephones, portable data processingdevices or portable web browsers.

Furthermore elements of a public land mobile network (PLMN) may alsocomprise video codecs as described above.

In general, the various embodiments of the invention may be implementedin hardware or special purpose circuits, software, logic or anycombination thereof. For example, some aspects may be implemented inhardware, while other aspects may be implemented in firmware or softwarewhich may be executed by a controller, microprocessor or other computingdevice, although the invention is not limited thereto. While variousaspects of the invention may be illustrated and described as blockdiagrams, flow charts, or using some other pictorial representation, itis well understood that these blocks, apparatuses, systems, techniquesor methods described herein may be implemented in, as non-limitingexamples, hardware, software, firmware, special purpose circuits orlogic, general purpose hardware or controller or other computingdevices, or some combination thereof.

The embodiments of this invention may be implemented by computersoftware executable by a data processor of the mobile device, such as inthe processor entity, or by hardware, or by a combination of softwareand hardware. Further in this regard it should be noted that any blocksof the logic flow as in the Figures may represent program steps, orinterconnected logic circuits, blocks and functions, or a combination ofprogram steps and logic circuits, blocks and functions. The software maybe stored on such physical media as memory chips, or memory blocksimplemented within the processor, magnetic media such as hard disk orfloppy disks, and optical media such as for example DVD and the datavariants thereof, CD.

The various embodiments of the invention can be implemented with thehelp of computer program code that resides in a memory and causes therelevant apparatuses to carry out the invention. For example, a terminaldevice may comprise circuitry and electronics for handling, receivingand transmitting data, computer program code in a memory, and aprocessor that, when running the computer program code, causes theterminal device to carry out the features of an embodiment. Yet further,a network device may comprise circuitry and electronics for handling,receiving and transmitting data, computer program code in a memory, anda processor that, when running the computer program code, causes thenetwork device to carry out the features of an embodiment.

The memory may be of any type suitable to the local technicalenvironment and may be implemented using any suitable data storagetechnology, such as semiconductor-based memory devices, magnetic memorydevices and systems, optical memory devices and systems, fixed memoryand removable memory. The data processors may be of any type suitable tothe local technical environment, and may include one or more of generalpurpose computers, special purpose computers, microprocessors, digitalsignal processors (DSPs) and processors based on multi-core processorarchitecture, as non-limiting examples.

Embodiments of the inventions may be practiced in various componentssuch as integrated circuit modules. The design of integrated circuits isby and large a highly automated process. Complex and powerful softwaretools are available for converting a logic level design into asemiconductor circuit design ready to be etched and formed on asemiconductor substrate.

Programs, such as those provided by Synopsys Inc., of Mountain View,Calif. and Cadence Design, of San Jose, Calif. automatically routeconductors and locate components on a semiconductor chip using wellestablished rules of design as well as libraries of pre-stored designmodules. Once the design for a semiconductor circuit has been completed,the resultant design, in a standardized electronic format (e.g., Opus,GDSII, or the like) may be transmitted to a semiconductor fabricationfacility or “fab” for fabrication.

FIG. 4 illustrates a flowchart of an encoding method according to anembodiment. The embodiment comprises steps for encoding into a bitstreaman indication that motion fields are stored, but only for inter-layermotion prediction (1016); encoding into a bitstream an indication on alimited scope of motion field usage (1018); encoding into a bitstream anindication whether or not to use the motion field for prediction (1020);encoding into a bitstream an indication of storage parameters forstoring motion information (1022). One or more of the previous steps canbe performed.

The foregoing description has provided by way of exemplary andnon-limiting examples a full and informative description of theexemplary embodiment of this invention. However, various modificationsand adaptations may become apparent to those skilled in the relevantarts in view of the foregoing description, when read in conjunction withthe accompanying drawings and the appended claims. However, all such andsimilar modifications of the teachings of this invention will still fallwithin the scope of this invention.

According to an example, there is provided a method comprising at leastone of the following:

-   -   a) encoding into a bitstream an indication that motion fields        are stored, but only for inter-layer motion prediction;    -   b) encoding into a bitstream an indication on a limited scope of        motion field usage;    -   c) encoding into a bitstream an indication whether or not to use        the motion field for prediction;    -   d) encoding into a bitstream an indication of storage parameters        for storing motion information.

According to a first embodiment, a step a) comprises two or more of thefollowing:

-   -   i. encoding into a bitstream an indication whether or not motion        fields are used within a layer for temporal motion vector        prediction;    -   ii. encoding into a bitstream an indication whether or not        inter-layer motion prediction is allowed to be used;    -   iii. encoding into a bitstream an indication whether or not        diagonal motion prediction is allowed to be used.

According to a second embodiment, in step b) the limited scope defineseither certain temporal sub-layers or picture types or both.

According to a third embodiment, step c) comprises using a specificalgorithms for inferring motion fields to be used for prediction.

According to a fourth embodiment, step c) comprises encoding in thebitstream a command or a syntax element for controlling motion fieldmarking.

According to a fifth embodiment, step d) comprises indicating eitherspatial resolution or accuracy of storing motion information.

According to a sixth embodiment, step d) comprises indicating whichparameters of the motion information are needed in the motionprediction.

According to a seventh embodiments, step d) comprises indicatingconstraints on parameter of the motion information which reduces thestorage space for motion fields.

According to a second example, there is provided an apparatus comprisingat least one processor and at least one memory including computerprogram code, the at least one memory and the computer program codeconfigured to, with the at least one processor, cause the apparatus toperform at least one of the following:

-   -   a) encode into a bitstream an indication that motion fields are        stored, but only for inter-layer motion prediction;    -   b) encode into a bitstream an indication on a limited scope of        motion field usage;    -   c) encode into a bitstream an indication whether or not to use        the motion field for prediction;    -   d) encode into a bitstream an indication of storage parameters        for storing motion information.

According to a third example there is provided a computer programproduct including one or more sequences of one or more instructionswhich, when executed by one or more processors, cause an apparatus to atleast perform at least one of the following:

-   -   a) encoding into a bitstream an indication that motion fields        are stored, but only for inter-layer motion prediction;    -   b) encoding into a bitstream an indication on a limited scope of        motion field usage;    -   c) encoding into a bitstream an indication whether or not to use        the motion field for prediction;    -   d) encoding into a bitstream an indication of storage parameters        for storing motion information.

According to a fourth example, there is provided a method comprising atleast one of the following:

-   -   a) decoding from a bitstream an indication that motion fields        are stored, but only for inter-layer motion prediction;    -   b) decoding from a bitstream an indication on a limited scope of        motion field usage;    -   c) decoding from a bitstream an indication whether or not to use        the motion field for prediction;    -   d) decoding from a bitstream an indication of storage parameters        for storing motion information.

According to a fifth example, there is provided an apparatus comprisingat least one processor and at least one memory including computerprogram code, the at least one memory and the computer program codeconfigured to, with the at least one processor, cause the apparatus toperform at least one of the following:

-   -   a) decode from a bitstream an indication that motion fields are        stored, but only for inter-layer motion prediction;    -   b) decode from a bitstream an indication on a limited scope of        motion field usage;    -   c) decode from a bitstream an indication whether or not to use        the motion field for prediction;    -   d) decode from a bitstream an indication of storage parameters        for storing motion information.

According to a sixth example there is provided a computer programproduct including one or more sequences of one or more instructionswhich, when executed by one or more processors, cause an apparatus to atleast perform at least one of the following:

-   -   a) decoding from a bitstream an indication that motion fields        are stored, but only for inter-layer motion prediction;    -   b) decoding from a bitstream an indication on a limited scope of        motion field usage;    -   c) decoding from a bitstream an indication whether or not to use        the motion field for prediction;    -   d) decoding from a bitstream an indication of storage parameters        for storing motion information.

1. A method comprising at least the following: decoding from a bitstreama first indication indicating whether or not one or more motion fieldsof a current picture associated with a layer are used for temporalmotion vector prediction of another picture in said layer; decoding fromthe bitstream a second indication indicating whether or not inter-layermotion prediction is used between said layer and another layer, whereininter-layer motion prediction provides for a motion prediction betweenpictures associated with different layers of a same time instant;storing, in memory, the one or more motion fields of the current picturefor temporal motion vector prediction of another picture in said layeror for inter-layer motion prediction; and when the second indicationindicates that inter-layer motion prediction is used between said layerand another layer, and the first indication indicates that the one ormore motion fields of the layer is/are not for temporal motion vectorprediction of another picture in said layer, deallocating memory used tostore the one or more motion fields of the current picture associatedwith said layer after all pictures from other layers of the same timeinstant using inter-layer motion prediction from said layer have beendecoded.
 2. A method according to claim 1 wherein deallocating memorycomprises marking the one or more motion fields of the current pictureassociated with said layer after all pictures from other layers of thesame time instant using inter-layer motion prediction from said layerhave been decoded as not being needed for prediction.
 3. A methodaccording to claim 1 wherein deallocating memory comprises using thememory previously used to store the one or more motion fields of thecurrent picture to store another motion field.
 4. A method according toclaim 1 further comprising receiving an indication of a maximum numberof motion fields that are needed for prediction that able to be storedin the memory.
 5. A method according to claim 1, wherein the storing, inmemory, the one or more motion fields of the current picture isperformed for N×N blocks, where N is larger than the minimum size of aprediction block in the current picture.
 6. A method according to claim1, further comprising: decoding an enhancement picture using inter-layermotion prediction from the stored one or more motion fields of thecurrent picture.
 7. A method according to claim 6, wherein the decodingthe enhancement picture further comprises: upsampling the stored one ormore motion fields of the current picture based on the sizes of thecurrent picture and enhancement picture.
 8. An apparatus comprising atleast one processor and at least one memory including computer programcode, the at least one memory and the computer program code configuredto, with the at least one processor, cause the apparatus to perform atleast the following: decode from a bitstream a first indicationindicating whether or not one or more motion fields of a current pictureassociated with a layer are used for temporal motion vector predictionof another picture in said layer; decode from the bitstream a secondindication indicating whether or not inter-layer motion prediction isused between said layer and another layer, wherein inter-layer motionprediction provides for a prediction between pictures associated withdifferent layers of a same time instant; store, in said at least onememory, the one or more motion fields of the current picture fortemporal motion vector prediction of another picture in said layer orfor inter-layer motion prediction; and when the second indicationindicates that inter-layer motion prediction is used between said layerand another layer, and the first indication indicates that the one ormore motion fields of the layer is/are not for temporal motion vectorprediction of another picture in said layer, deallocate memory used tostore the one or more motion fields of the current picture associatedwith said layer after all pictures from other layers of the same timeinstant using inter-layer motion prediction from said layer have beendecoded.
 9. An apparatus according to claim 8 wherein the apparatus iscaused to deallocate memory by marking the one or more motion fields ofthe current picture associated with said layer after all pictures fromother layers of the same time instant using inter-layer motionprediction from said layer have been decoded as not being needed forprediction.
 10. An apparatus according to claim 8 wherein the apparatusis caused to deallocate memory by using the memory previously used tostore the one or more motion fields of the current picture to storeanother motion field.
 11. An apparatus according to claim 8 wherein theapparatus is embodied by a mobile device.
 12. An apparatus according toclaim 8, wherein the at least one memory and the computer program codeare configured to, with the at least one processor, cause the apparatusto store, in memory, the one or more motion fields of the currentpicture for N×N blocks, where N is larger than the minimum size of aprediction block in the current picture.
 13. An apparatus according toclaim 8, wherein the at least one memory and the computer program codeare further configured to, with the at least one processor, cause theapparatus to: decode an enhancement picture using inter-layer motionprediction from the stored one or more motion fields of the currentpicture.
 14. An apparatus according to claim 13, wherein the at leastone memory and the computer program code are configured to, with the atleast one processor, cause the apparatus to decode the enhancementpicture by: upsampling the stored one or more motion fields of thecurrent picture based on the sizes of the current picture andenhancement picture.
 15. A computer program product comprising at leastone non-transitory computer-readable storage medium including one ormore sequences of one or more instructions which, when executed by oneor more processors, cause an apparatus to at least perform at least thefollowing: decoding from a bitstream a first indication indicatingwhether or not one or more motion fields of a current picture associatedwith a layer are used for temporal motion vector prediction of anotherpicture in said layer; decoding from the bitstream a second indicationindicating whether or not inter-layer motion prediction is used betweensaid layer and another layer, wherein inter-layer motion predictionprovides for a prediction between pictures associated with differentlayers of a same time instant; storing, in memory, the one or moremotion fields of the current picture for temporal motion vectorprediction of another picture in said layer or for inter-layer motionprediction; and when the second indication indicates that inter-layermotion prediction is used between said layer and another layer, and thefirst indication indicates that the one or more motion fields of thelayer is/are not for temporal motion vector prediction of anotherpicture in said layer, deallocating memory used to store the one or moremotion fields of the current picture associated with said layer afterall pictures from other layers of the same time instant usinginter-layer motion prediction from said layer have been decoded.
 16. Acomputer program product according to claim 15 wherein the apparatus iscaused to deallocate memory by marking the one or more motion fields ofthe current picture associated with said layer after all pictures fromother layers of the same time instant using inter-layer motionprediction from said layer have been decoded as not being needed forprediction.
 17. A computer program product according to claim 15 whereinthe apparatus is caused to deallocate memory by using the memorypreviously used to store the one or more motion fields of the currentpicture to store another motion field.
 18. A computer program productaccording to claim 15, wherein the one or more sequences of one or moreinstructions which, when executed by one or more processors, cause theapparatus to store, in memory, the one or more motion fields of thecurrent picture cause storage, in the memory, of the one or more motionfields of the current picture to be performed for N×N blocks, where N islarger than the minimum size of a prediction block in the currentpicture.
 19. A computer program product according to claim 15, whereinthe one or more sequences of one or more instructions which, whenexecuted by one or more processors, further cause the apparatus to:decode an enhancement picture using inter-layer motion prediction fromthe stored one or more motion fields of the current picture.
 20. Acomputer program product according to claim 19, wherein the one or moresequences of one or more instructions which, when executed by one ormore processors, cause the apparatus to decode the enhancement picturecomprise one or more sequences of one or more instructions which, whenexecuted by one or more processors, cause the apparatus to: upsample thestored one or more motion fields of the current picture based on thesizes of the current picture and enhancement picture.
 21. A methodcomprising at least the following: encoding into a bitstream a firstindication indicating whether or not one or more motion fields of acurrent picture associated with a layer are used for temporal motionvector prediction of another picture in said layer; encoding into thebitstream a second indication indicating whether or not inter-layermotion prediction is used between said layer and another layer, whereininter-layer motion prediction provides for a prediction between picturesassociated with different layers of a same time instant; and storing, inmemory, the one or more motion fields of the current picture fortemporal motion vector prediction of another picture in said layer orfor inter-layer motion prediction, wherein when the second indicationindicates that inter-layer motion prediction is used between said layerand another layer, and the first indication indicates that the one ormore motion fields of the layer is/are not for temporal motion vectorprediction of another picture in said layer, permit memory used to storethe one or more motion fields of the current picture associated withsaid layer to be deallocated after all pictures from other layers of thesame time instant using inter-layer motion prediction from said layerhave been decoded.
 22. A method according to claim 21 further comprisingencoding into the bitstream an indication of a maximum number of motionfields that are needed for prediction that able to be stored in thememory.
 23. A method according to claim 21, wherein the storing, inmemory, the one or more motion fields of the current picture isperformed for N×N blocks, where N is larger than the minimum size of aprediction block in the current picture.
 24. A method according to claim21, further comprising: encoding an enhancement picture usinginter-layer motion prediction from the stored one or more motion fieldsof the current picture.
 25. A method according to claim 24, wherein theencoding the enhancement picture further comprises: upsampling thestored one or more motion fields of the current picture based on thesizes of the current picture and enhancement picture.
 26. An apparatuscomprising at least one processor and at least one memory includingcomputer program code, the at least one memory and the computer programcode configured to, with the at least one processor, cause the apparatusto perform at least the following: encode into a bitstream a firstindication indicating whether or not one or more motion fields of acurrent picture associated with a layer are used for temporal motionvector prediction of another picture in said layer; encode into thebitstream a second indication indicating whether or not inter-layermotion prediction is used between said layer and another layer, whereininter-layer motion prediction provides for a prediction between picturesassociated with different layers of a same time instant; and store, inmemory, the one or more motion fields of the current picture fortemporal motion vector prediction of another picture in said layer orfor inter-layer motion prediction, wherein the second indicationindicates that inter-layer motion prediction is used between said layerand another layer, and the first indication indicates that the one ormore motion fields of the layer is/are not for temporal motion vectorprediction of another picture in said layer, permit memory used to storethe one or more motion fields of the current picture associated withsaid layer to be deallocated after all pictures from other layers of thesame time instant using inter-layer motion prediction from said layerhave been decoded.
 27. An apparatus according to claim 26 wherein theapparatus is further caused to encode into the bitstream an indicationof a maximum number of motion fields that are needed for prediction thatable to be stored in the memory.
 28. An apparatus according to claim 26,wherein the at least one memory and the computer program code areconfigured to, with the at least one processor, cause the apparatus tostore, in memory, the one or more motion fields of the current picturefor N×N blocks, where N is larger than the minimum size of a predictionblock in the current picture.
 29. An apparatus according to claim 26,wherein the at least one memory and the computer program code arefurther configured to, with the at least one processor, cause theapparatus to: encode an enhancement picture using inter-layer motionprediction from the stored one or more motion fields of the currentpicture.
 30. An apparatus according to claim 29, wherein the at leastone memory and the computer program code are configured to, with the atleast one processor, cause the apparatus to encode the enhancementpicture by: upsampling the stored one or more motion fields of thecurrent picture based on the sizes of the current picture andenhancement picture.
 31. An apparatus according to claim 26 wherein theapparatus is embodied by a mobile device.
 32. A computer program productcomprising at least one non-transitory computer-readable storage mediumincluding one or more sequences of one or more instructions which, whenexecuted by one or more processors, cause an apparatus to at leastperform at least the following: encoding into a bitstream a firstindication indicating whether or not one or more motion fields of acurrent picture associated with a layer are used for temporal motionvector prediction of another picture in said layer; encoding into thebitstream a second indication indicating whether or not inter-layermotion prediction is used between said layer and another layer, whereininter-layer motion prediction provides for a prediction between picturesassociated with different layers of a same time instant; and storing, inmemory, the one or more motion fields of the current picture fortemporal motion vector prediction of another picture in said layer orfor inter-layer motion prediction, wherein the second indicationindicates that inter-layer motion prediction is used between said layerand another layer, and the first indication indicates that the one ormore motion fields of the layer is/are not for temporal motion vectorprediction of another picture in said layer, permit memory used to storethe one or more motion fields of the current picture associated withsaid layer to be deallocated after all pictures from other layers of thesame time instant using inter-layer motion prediction from said layerhave been decoded.
 33. A computer program product according to claim 32,wherein the one or more sequences of one or more instructions which,when executed by one or more processors, cause the apparatus to store,in memory, the one or more motion fields of the current picture causestorage, in the memory, of the one or more motion fields of the currentpicture to be performed for N×N blocks, where N is larger than theminimum size of a prediction block in the current picture.
 34. Acomputer program product according to claim 32, wherein the one or moresequences of one or more instructions which, when executed by one ormore processors, further cause the apparatus to: encode an enhancementpicture using inter-layer motion prediction from the stored one or moremotion fields of the current picture.
 35. A computer program productaccording to claim 34, wherein the one or more sequences of one or moreinstructions which, when executed by one or more processors, cause theapparatus to encode the enhancement picture comprise one or moresequences of one or more instructions which, when executed by one ormore processors, cause the apparatus to: upsample the stored one or moremotion fields of the current picture based on the sizes of the currentpicture and enhancement picture.